18.2.2.1 Sample Data

my_html = '''
<div id="my-id1" class='title'> 
    <p>This Is My Title</p>
    
    <div id="my-id2" class='subtitle' custom_attr='funny'>
        <p>This is Subtitle</p>
    </div>
    
    <div id="my-id3" class='title',   custom_attr='funny'>
        <p>This is paragraph1</p>
        <p>This is paragraph2</p>
        <h3>This is paragraph3</h3>
    </div>
</div>
'''
soup = BeautifulSoup(my_html)

18.2.2.2 First Match

ID Selector
Everthing under the selected tag will be returned.

soup.find(id='my-id1')

#:> <div class="title" id="my-id1">
#:> <p>This Is My Title</p>
#:> <div class="subtitle" custom_attr="funny" id="my-id2">
#:> <p>This is Subtitle</p>
#:> </div>
#:> <div class="title" custom_attr="funny" id="my-id3">
#:> <p>This is paragraph1</p>
#:> <p>This is paragraph2</p>
#:> <h3>This is paragraph3</h3>
#:> </div>
#:> </div>

Class Selector

soup.find(class_='subtitle')

#:> <div class="subtitle" custom_attr="funny" id="my-id2">
#:> <p>This is Subtitle</p>
#:> </div>

Attribute Selector

soup.find(custom_attr='funny')

#:> <div class="subtitle" custom_attr="funny" id="my-id2">
#:> <p>This is Subtitle</p>
#:> </div>

soup.find(       custom_attr='funny')

#:> <div class="subtitle" custom_attr="funny" id="my-id2">
#:> <p>This is Subtitle</p>
#:> </div>

soup.find('div', custom_attr='funny')

#:> <div class="subtitle" custom_attr="funny" id="my-id2">
#:> <p>This is Subtitle</p>
#:> </div>

18.2.2.3 Find All Matches

find_all

soup = BeautifulSoup(my_html)
multiple_result = soup.find_all(class_='title')
print( 'Item 0: \n',     multiple_result[0],
       '\n\nItem 1: \n', multiple_result[1])

#:> Item 0: 
#:>  <div class="title" id="my-id1">
#:> <p>This Is My Title</p>
#:> <div class="subtitle" custom_attr="funny" id="my-id2">
#:> <p>This is Subtitle</p>
#:> </div>
#:> <div class="title" custom_attr="funny" id="my-id3">
#:> <p>This is paragraph1</p>
#:> <p>This is paragraph2</p>
#:> <h3>This is paragraph3</h3>
#:> </div>
#:> </div> 
#:> 
#:> Item 1: 
#:>  <div class="title" custom_attr="funny" id="my-id3">
#:> <p>This is paragraph1</p>
#:> <p>This is paragraph2</p>
#:> <h3>This is paragraph3</h3>
#:> </div>

CSS Selector using select()

Above can be achieved using css selector. It return an array of result (multiple matches).

multiple_result = soup.select('.title')
print( 'Item 0: \n',     multiple_result[0],
       '\n\nItem 1: \n', multiple_result[1])

#:> Item 0: 
#:>  <div class="title" id="my-id1">
#:> <p>This Is My Title</p>
#:> <div class="subtitle" custom_attr="funny" id="my-id2">
#:> <p>This is Subtitle</p>
#:> </div>
#:> <div class="title" custom_attr="funny" id="my-id3">
#:> <p>This is paragraph1</p>
#:> <p>This is paragraph2</p>
#:> <h3>This is paragraph3</h3>
#:> </div>
#:> </div> 
#:> 
#:> Item 1: 
#:>  <div class="title" custom_attr="funny" id="my-id3">
#:> <p>This is paragraph1</p>
#:> <p>This is paragraph2</p>
#:> <h3>This is paragraph3</h3>
#:> </div>

More granular exmaple of css selector.

soup.select('#my-id1 div.subtitle')

#:> [<div class="subtitle" custom_attr="funny" id="my-id2">
#:> <p>This is Subtitle</p>
#:> </div>]

Using contains()

soup.select("p:contains('This is paragraph')")

#:> [<p>This is paragraph1</p>, <p>This is paragraph2</p>]

Combining ID, Class and Custom Attribute in the selector

soup.select("div#my-id3.title[custom_attr='funny']:contains('This is paragraph')")

#:> [<div class="title" custom_attr="funny" id="my-id3">
#:> <p>This is paragraph1</p>
#:> <p>This is paragraph2</p>
#:> <h3>This is paragraph3</h3>
#:> </div>]

18.2.4.1 Get Content get_text(strip=, separator=)

• Use strip=True to strip whitespace from the beginning and end of each bit of text
  
• Use `separator=‘\n’ to specify a string to be used to join the bits of text together
• It is recommended to use strip=True, separator='\n' so that result from different operating system will be consistant

soup = BeautifulSoup(my_html)
elem = soup.find(id = "my-id3")
elem.get_text(strip=False)

#:> '\nThis is paragraph1\nThis is paragraph2\nThis is paragraph3\n'

• strip=True combine with separator will retain only the user readable text portion of each tag, with separator seperating them

elem.get_text(strip=True, separator='\n')

#:> 'This is paragraph1\nThis is paragraph2\nThis is paragraph3'

18.2.4.2 Splitting Content

It is useful to split using separator into list of string.

elem = soup.find(id = "my-id3")
elem.get_text(strip=True, separator='\n').split('\n')

#:> ['This is paragraph1', 'This is paragraph2', 'This is paragraph3']

18.2.5.1 Get The Element

elems = soup.select("div#my-id3.title[custom_attr='funny']:contains('This is paragraph')")
elem = elems[0]
elem

#:> <div class="title" custom_attr="funny" id="my-id3">
#:> <p>This is paragraph1</p>
#:> <p>This is paragraph2</p>
#:> <h3>This is paragraph3</h3>
#:> </div>

18.2.5.2 Traversing Children

All Children In List findChildren()

elem.findChildren()

#:> [<p>This is paragraph1</p>, <p>This is paragraph2</p>, <h3>This is paragraph3</h3>]

Next Children findNext()

• If the element has children, this will get the immediate child
  
• If the element has no children, this will find the next element in the hierechy

first_child = elem.fin
print( 
elem.findNext().get_text(strip=True), '\n', 
elem.findNext().findNext().get_text(strip=True), '\n')

#:> This is paragraph1 
#:>  This is paragraph2

18.2.5.3 Traversing To Parent parent()

elem_parent = elem.parent
elem_parent.attrs

#:> {'id': 'my-id1', 'class': ['title']}

18.2.5.4 Get The Sibling findPreviousSibling()

Sibling is element at the same level of hierachy

elem_prev_sib = elem.findPreviousSibling()
elem_prev_sib.attrs

#:> {'id': 'my-id2', 'class': ['subtitle'], 'custom_attr': 'funny'}