Each and every web site has a reason for its existence.
Whatever the reason for a site's existence it needs people to visit and interact with the site in order for the site to achieve its purpose. In order for people to participate in the site, the site needs to provide those people with something that they are looking for and the people need to be able to find the site.
There is little point in having a site that is easy to find if it doesn't provide the people who find it with what they are looking for. Those people will just leave again straight away and you will not achieve the purpose that is the reason for the site existing with respect to those people.
So what does all this have to do with search engine optimisation? Well a search engine is a web site and it to has a reason for its existence. So like any other site it needs to present people with what they are looking for. What people are looking for a search engine to provide for them is links to pages that contain what they are looking for. If a search engine doesn't give them that then those people will have wasted their time using that search engine and will look for alternative ways to find what they are looking for. In order for the search engine to satisfy the reason for its existence (which is generally to make money for the owners) it has to provide links to web pages that are relevant to what is being searched for. If the results are not relevant then people will not use the search engine and the owners of the search engine will not make money.
It is the individual visitors who know what it is that they are looking for and what they type into the search engine is the best that they can do in converting that into words. The search engine then has to interpret the words that are entered and try to determine which pages that it knows about are the most likely ones to contain what that person is looking for based on what was typed in and what the search engine knows about the pages. This matching process is made more complicated both by the fact that what the person is really looking for may or may not be adequately indicated by the words they entered and also by the fact that the search engine is not really intelligent and so only has a vague idea of what a page is about based on its interpretation of the content of the page. If search engines were able to accurately determine exactly what every web page was about then the search engine would be able to provide links to the pages that are the ones that are most likely to contain the information that the person is looking for - assuming that the search engine also correctly interpreted the meaning of what the person typed in. It is because the search engines are not smart enough to determine exactly what any given web page is about that you need to modify the web page slightly in order to make it easier for the search engines to work out what your page is saying. Of course making the page easier to understand will also make it easier for your less intelligent visitors to work out what the page is about as well. So any legitimate changes that are made inside the web page in order to optimise it for the search engines is in fact just improving the readability and understandability of your content. Any modifications that you make to the page content just for the search engines that does not also help those visiting your page may result in your page being easier to find but it will also mean that it is now less likely to provide what people are looking for as you have made the page less friendly to real people. Doing that therefore defeats the purpose in having the web site in the first place. So any changes that you make in the web page itself are either being made to help real people as well as search engines (and so are not specifically SEO) or if they are strictly SEO changes then their impact will be to increase the number of people who find your page but decrease the number of people who use your page for its intended purpose. So in page SEO is definitely something to avoid.
The search engines are not the only way to find a web page. Some people find one web site by following a link from another related web site. By providing links to other sites that contain related material that is not directly in competition with your site you make your site more useful as a gateway for people to use whenever they want anything that is either directly or indirectly related to your site. This makes them more likely to use your site as the starting point when they want something that is related to what your site is about rather than starting from a search engine. While people starting from your site in order to get to the related site you have linked to that contains the what they are looking for does not directly achieve your purpose on this occasion it does mean that those visitors are used to visiting your site and so when they or someone they know does need what your site offers then they will not be using a search and ending up with one of your competitors. Your site therefore benefits by providing links to related sites. Since this applies to all web sites it means that other sites that cover related topics will also benefit by providing links to your site. This will provide you with more visitors to your site who have got there without having found your page via a search engine. The value of these links between sites is determined by the relevance of what the page being linked to has with respect to what the page or site containing the link is about. If the topics are sufficiently related then the link will be useful and both sites will benefit. If the topics are not related then no one will follow the link and so the site being linked to will not benefit and the site providing the link has made itself less useful by wasting space on links that no one is interested in.
While search engines do take into account the number and relevance of links to a web page in determining its importance, they only do so because those links are useful and relevant to real people and the more such incoming links that a page has, the more other sites there are where the owner has determined that they can gain value for their own site by linking to that page because of the benefits mentioned above. If there were no benefit to be gained by the site providing the link then there would be no reason for the search engines to take the link into account. Since search engines do not have precise knowledge on what the pages are about they don't really know for certain how relevant one topic is to the other and so they have to judge based on what they do know regarding those links. Creating lots of links between pages on two specific unrelated topics may lead to the search engine deciding that the two topics are related however real people will not be following those links and if the pages on the unrelated topic show up in the search results the person searching will not find what they want. The search engines will therefore want to resolve any such mis-impressions as quickly as possible. Nothing that you do with regard to links either from your site to others or with asking others to link to you will only impact on search results. Only those links that would still be useful if search engines didn't exist will be useful with respect to the search engines. There is therefore no connection between SEO and link building.
So just what is there that is left that might possibly be of benefit with respect to search that doesn't directly provide benefits to real people as well? Well there is the information that the search engine itself displays about the page in the search results. The better that description matches to what the person is searching for, the more likely that they are to actually click on that link and visit the page. The more people that click on a link in the results and who don't return to the search results relatively quickly, the more likely it is that the person has found what they are looking for. This makes that particular link more valuable to the search engine as one that should be displayed toward the top of the results for that particular search term. Just what the search engine pulls out of the web page to display as its summary of the page varies depending on the web browser. Most used to simply use the title and meta description in their results page. As the meta description isn't really used anywhere else it is about the only thing that could be changed that directly related to SEO. Search engines used to use the meta keywords tag as well in trying to determine page relevance however abuse of the content of that tag means that it is now ignored. Similar abuse of the meta description tag means that some search engines now use a part of the actual page content in place of the meta description.
This means that making sure that your page is arranged in such a way that the search engines will extract an appropriate description of your page for their results while still making sure that the page reads well for the real people who end up reaching it is the only change that only benefits the search engines and which doesn't benefit any real people directly. This change is therefore the only one that therefore deserves to be referred to as Search Engine Optimisation as it is the only change that is not being made to directly provide benefits to your visitors. On second thoughts this change would probably also make the content friendlier to the real people visiting.
Based on the above, there isn't anything that represents Search Engine Optimisation that isn't also optimising for real people. If you have already optimised for the real people you hope will visit the page then there isn't anything left to do to optimise it for search engines that will not result in a worse experience for your real visitors and which will therefore eventually defeat the purpose you have in optimising for search engines in the first place.