Statistics includes many methods and some machine learning algorithms could overlap the statistical methods such as hypothesis testing could be applied by some forms of again mathematics and statistics because we know machine learning and data science tries to pass over statistics or update it. Some methods don’t change for example dripping or fast run, or some tactics on sports. They are legit and fundamental. However, some things should be updated like forwarding tactics, defensive tactics game theory related conditions. Regression methods that Ridge and other methods could surpass the complex machine learning algorithms. Frankly, a statistician with kaggle experience being grandmaster should explain these factors but they seem like a pro gamble or soccer players that don’t stand on theory or business generally theoretical problems solving which is hard. However, with the fact of this, Kaggle may not be a profitable place for newbies or utility of competitions. You can use other regression models in your business and they may be more robust than explaining these models rather than explaining complex machine learning models.
Linear Regression
Linear regression could be seen as a linear line passing through middle imaginary points between two or more variables and shows the direction of the points. That’s it is. That’s calculated more complex than expressing it but it is rough could de be described as that. If we see the formula
We sum up the differences of the dependent variable from its mean multiply it applying the same process to a dependent variable then divide it with again dependent variables square. We can think of this process as normalization. Probably we are getting to multiply in multiples regression on the upper side all the sum of differences from mean and bottom again will be the other dependent variables multiplying yet we will get total multiplications of all again. This β is a factor and we find that as it is and we can create a better formula for this eventually. As we can see, improvement of this logic on batches of smaller data and applying more detailed information on the dataset could have resulted in more advanced regression techniques. Other techniques or machine learning models calculations could be seen from that perspective.
Logistic Regression
Logistic Regression may be received as a binary classification algorithm in a simple term. It classifies the features of target value and classifies it according to these features. Logistic Regression function logic is very solid and it could be applicable for all binary categorization problems because of its simple design.
First assumption is that, l = logk(p/p-1) = b + b1x1 + b2x2 + b3x3 our features and there so when we get these probability values into log we can see that our values changes between 0 and 1. Here, the formula changes into p = k^(b+b1x1+b2x2…)/k^(b+b1x1+b2x2.. + 1) so that we reach the p = 1/1 + k^-(b+b1x2+b2x2..) so that if we give e natural 2.7… number to we reach the last formula of logistic regression.
After that, we calculate this b, b1, and other features parameters for optimized k^(N) value that N will be some number so you can create an optimization learning rate or gradient system for that number on machine learning system or you can develop a more reliable formula for upper value or general logistic regression. That way, we train our parameters with the results of the values then we predict future events from that
For example, we have made an A/B test and we have analytics metrics of two pages and these metrics are evaluated for all the A/B test pages. Therefore, we can create a table of which parameters we select and which we don’t after then, we can automate the selection with logistic regression nicely. Another thing we can do is maybe understanding product buying intention could be that is extracted from product metrics. That we can say From User ID, this user probably won’t buy the product or the user will buy it with the right metrics
Ridge Regression
The ridge regression is used when the high correlation is there between independent variables meaning features. Ridge regression is great for blocking overfitting and it seems very useful for real data.
This is the formula of ridge regression. We are defining the parameters of independent variables by minimizing the error rate of the results. We can change this formula and create another mathematical formula better than that with these parameters. It is up to you. We can think of a linear regression line as a new coordinate system, shift the coordinate system and create polynomial regression with on this direction. That way, we can minimize more the error. Minimizing the Error rate so low on tests with mathematical assumptions could lead to another machine learning algorithm from this perspective. You can write this method on your algorithm again adding learning rate to parameters and add each iteration and batch with gradient. That way, when we reach the last optimized value which does not change, it is meant that this parameter is our determined parameters.
This is a site I found for diverse statistics explanation and broad subjects about again statistics. I focus on statistics because it is abstract and it was mainly the hardest lecture at my University. We see that Gauss as a Mathematician first started this area as I know and then it is got developed by many academics and scientist. Statistics is better for math weighted people in computer science or programmatic area and it could open new ways of thinking on your imagination.
Tikhonov Regularization
I heard this statistician before maybe but here while I am looking on Wikipedia, it seems very significant result I see about ridge regression. Ridge regression is mentioned as sub implementation of “Tikonov Regularization”. Well, the generalizations must be important rather than sub formulas and states. I think being a statistics wizard gets you so much power when it comes to business and academic-related issues generally. Therefore, you could process the mathematics of it and work on it by selecting related terms here
So we have Ax = b such a matrix equation and we have a matrix as A and we multiply it with x and we get b vector. We can find many solutions or no solutions so we try to minimize this min x∈Rn {kAx − bk 2 + kLµxk 2 equation. kAxk − bk ≤ ηε we can see this as minimization of error so kAxk − bk 2 = Xm j=k+1 eb 2. It is a long and laTeX wroten proof you should read it here. We see that sum of b/standard deviation minimization is the function we are aiming for. That is so minimal standard deviation beyond the data points and maximal correctness at points could be said that is the purpose. The general solution as mentioned before must be not overfitted for future values or past values.
Lasso regression and ridge regression is quite similar. The formula of lasso regression is
In lasso regression, the hypermeter alpha and coefficients are tried to minimize equivalently in ridge regression. The only real difference is that, rather than taking the sum of squares of weights, we take only absolute values of weights. That’s the difference between them. Ridge regression works well if there are correlated similar values columns in the data frame. Lasso regression is great for regular data mostly. We can see that ridge regression penalizes weights more. That’s why we can see that weights could reach low values and not effective values from ridge regression in positive status. However, in negative and positive status, lasso could tend to fit well with the right positive and negative set of weights
Elastic Net Regression
Before we described two types of penalizes one is a * sum(|w|) and the other a * sum(w^2) type of errors. These are the l1 and l2 errors as described in the literature. These errors, when both of them are used in the minimization of error formula, it is described as elastic net regression. Therefore we thee formula as L = ∑( Ŷi– Yi)² + λ∑ β² + λ∑ |β| here the formula simply shows the weight calculations combination as a sum. That way we reach a smoother calculation and maybe better result in the end.
Cycolmatic complexity is a software quantative measure metric to calculate the source code complexity by analyzing number of linearly independent path on the program. It looks the nodes of the program and how the control flow is connected by nodes of the program. It could be applied into classes, functions, moduls of the program. The Cyclomatic Complexity looks for if else structure and their changes by observing node states. This is the formula of the metric M = E − N + 2P where E is the number of edges on the graph, N is the number of the nodes of the graph and P is the number of connected components. For example in this code we see a function and its application:
a = 5
b = 4
givenInt = int(input(“Give an integer:”))
if ( a > b):
if (a > givenInt):
print(“nice job)
else:
print(“work harder”)
else:
if(b > givenInt):
print(“Not bad”)
else:
print(“too bad”)
In such a program we can pattern the program such as:
if (c1())
f1();
else
f2();
if (c2())
f3();
else
f4();
Or we can see this code body as a node such
We can see that first on has a complexity of (edge:6, node:7, connected components:0) we can measure this as M = E – N + 2 so the complexity of the program is M = 6 – 7 + 2 = 1 for the result. Therefore other is (edge:9, node:7, connected components:1) = 9 – 7 + 2 = 4 for the result. Less chain on the program means less complexity. In brief, The cyclomatic complexity measures the decision tree of the program and decision structure. Memory and other issues are calculated by other metrics.
There are other complexity metrics we can see here such as branching complexity (Sneed Metric), data access complexity (Card Metric), data complexity (Chapin Metric), data flow complexity (Elshof Metric), decisional complexity (McClure Metric). In industry, mainly time complexity and space complexity metrics are measured to calculate complexity of a program for performance. These two metric, first one measures the loop system as Big(O) measurement system.
Big O Mathematical Notation
Big O notation is used to understand algorithm for efficiency. It looks for loops and its structure and progress of program by its assignment system. We know that short and capable program is valuable. Therefore, Big O measures the program and we can set increase capasity program and select best performance from the algorithms. There linear, constant, expontential and logarthmic complexities by the means of Big O measurement. Logarthmic complexity is the wished complexity in general. Because it converges a particular time value in the long run with data. Therefore, we can reach some constant complexity in limit probably. If we want to see a algorithm by example
A = 1
B = 2
For i in range(10):
A = B + 2
Print(A)
Here we see that two operations are made outside the loop then one operation with 10 loops inside a loop then one operation. We can see this as 1(N) + 3 for the result. N is here 10 so that result complexity is 13. It is a linear type complexity, if we put more value to loop, it will rise linearly by input.
Algorithmic Complexity
Algorithmic complexity is a term passed in algorithmic information theory that is interested in algorithm result data and its information value entropy. The algorithmic information theory is founded by Ray Solomonoff. For example, we create a string or a result from the algorithm. If the result is “01010101” the string could easily be generated by repeating “01”. However, if we result in such “011010110101010111000110” we cannot create a simple algorithm for this result. Therefore, from the result of the program, we analyze the algorithm of the program and we can reach a conclusion of the program and its capacity by the complexity or information value of the result. Therefore, we can simplify the algorithm by these measurements and put another simple program rather than the complex or medium size structures. It is also a valuable term by natural language processing or cryptology aspects.
What is an Algorithm?
An Algorithm is a list of instructions that a program operates. Algorithms are everywhere when we use instructions to complete our tasks. Algorithms are created to solve a programming problem and while there is a problem as medium or complex, we should generate an algorithm to solve it. Computer algorithms take input and output. For instance, deep learning algorithms take input and evaluate them in layers and nodes with w(input) + bias. That way they converge and calculate some values as output. Machine learning algorithms also take input as column values as rows and calculate them and create formulation from these inputs and show and output formulation or calculation. Algorithms are used for the automation of mathematical operations and other processes of works in the business world. IBM could be seen as the first algorithm implementation business company by creating time clocks and punch cards. Before IBM, the automation and generation of all the systems in human history could be seen as an algorithm. In Roma history, the war tactics and their systematic progress could be seen as an algorithm process. That way, we can see that, all the systematic reactive operations as algorithmic systems.
As we can see, the time complexity is measured like this and the general perspective of algorithm and program complexity are briefly these concepts. Besides, we can see the algorithm structures, how they work and what is their progress in the program
General Artificial Intelligence Concept and Its Broad Subjects and Relative Applicaitons
Artificial Intelligence is a broad subject and there are many academic aspects and topics about it. Today, many researchers and scientists working on these subjects, and their many academic applications of it. Machine learning is a subtopic of Artificial Intelligence and there are many related topics about it. Therefore, you can view this subject from different perspectives. On the other hand, applying Artificial Intelligence in business can narrow our knowledge and capabilities in the AI area. Therefore a business needs to filter the subject however, reaching general intelligence is the main goal of so many topics in the area that means manifesting a human-like creature. That is a hard topic and needs robotic perspective, neuroscience, and another area of professional expertise. Before corona, these areas were so active now after the disaster we can see another rise in these topics, and also these topics could seem problematic about AI and its relationship with humanity
One aspect could be said that understanding how the brain works and its functionality and describing it with mathematics. It is a broad subject you could define. Mathematical psychology and other concepts could define brain functions. Modeling the brain is a topic but modeling and engineering is another matter that could be discussed. We know that brain is a complex organ. Therefore, its functionality is not understood well by its users. It could be especially hard when we try to observe ourselves. Programming brain is also a hard concept to apply in real. AI is not an old concept but we can say it is old because of the haste of improvements. AI works with simple ideas and these simple ideas when well-engineered could lead to very prominent technologies
Casuality and Its Occurrence in Artificial Intelligence
Causality, cause and effect is an important factor in Artificial Intelligence. Some things should cause events and they should affect something. A particular AI system should understand this casualty. Generally, correlation and other systems result from some causation. Therefore, we can say that correlation could be seen as causation yet it could be not so. Thus, causation is a real term that wanted to be integrated with programs. A program that could understand the causation and copy the system of this could adapt and overcome the problems. Therefore, an AI system could copy and integrate with the system plus it should benefit and create cycles that are new and intelligent. It is a hard topic but with improvements in specific fields, the algorithms and solutions on problems will increase parallelly. Hence, AI will be understood more broadly and extensively. In an abstract system, we could create an observer that could extract results from data and conclude a resolution after. However, these resolutions could be biased. That is where reasoning skills and their broad definition comes. Therefore, the reasoning is a big part of Artificial systems
Knowledge Representation and Reasoning & Its Importance for factors of AI
Knowledge Representation is a field of AI. It is formed from brain and mind structure and resolves into algorithms and uses logic to represent relationships about states. Semantic nets and problem solvers could be a great example of this subject. I can describe the semantic relationship here. For instance, we represent {word1 -> word2, word3 -> word2, word4 -> word2, word3 -> word1} such system if we state the function of the word we can set a semantic net on the graph. We can relate these words with ontologies. Then an observer AI system could view the relationships and answer the queries base on them. The data structure base and memory of this structure, its application, and optimization on this could result in a modest AI application. That way computer science turns into computer engineering. For example, we can state a knowledge graph and nodes. From nodes, we can set functions for specific node types and we can generate a graph from these nodes and a fractal body could result. From this format, we can apply an adaptor system for this format and by applying humanistic data into these structures, semantic networks work and output could transform this data. A human brain with adaptors to nature probably working in this manner.
Open AI, Elon Musk’s Efforts on Artificial Intelligence, Conclusion to Neurolink and
Open AI is founded in 2015 by Elon Musk and Sam Altman after a while, Elon Musk decided that Artificial Intelligence is dangerous for humanity or its improvements around the world and he decided to leave this company and Open AI efforts have continued to this day after 2018. Elon Musk is very interested in AI topic and he seems connected to this subject from Asimov’s books and the robot laws about it. Unfortunately, in the end, he decided to continue with another system as Neurolink to represent human data on the computer to collect data and to find meaningful connections as scientific discovery. He is aiming to connect the human body as robotic assemble with machines as seen as cyborgs. He mentioned such statements before on Twitter so we can assume he set a mission about it. We can think that combining human and machine can exclude the human-robot conflict and creates an open way for AI revolution and human evolution. From that perspective, if we believe is that improvement is inevitable, this form of transformation could be seen so valid reasons. So there is a two systematic way that AI today thrives, one seems a path that Elon Musk personally investing as Neurolink, and GPT-3 system from Open-AI which is also the creator of Dota-2 wining bots and so much. Google and Facebook also created such great systems with reinforcement learning with machine learning applications. From business and idealistically speaking as I mention, Elon Musk puts a more strict line about this subject, and his footsteps on Artificial Intelligence topic creates wider value on the world today.
The Practise to Get Organic Traffic - Search Engine Optimization
Search Engine Optimization is a practice to get quantity and quality of traffic to website from organic search meaningly achieve traffic from search engine ranks. It is a broad subject and includes many topics from HTML, CSS, javascript, and frontend, backend functions. Year to year, search engine optimization tactics and functions improve while search engines especially Google algorithms improve. Therefore, Search Engine necessities getting complex while these algorithms developed with more capacity. Google and other search engines want some specific features to be improved on the sites. Generally content, links from other sites, the performance of the site, and its HTML structure goodness. By these facts, Google and other engines decide which website is the most valuable to shown in some query. That way one rank high gets the most traffic from these words. In this scenario, word volume could be important. For instance, a word with 20000 volume and 200000 volume can differ but ranking with the word with the volume of 20000 could be easier. That way, SEO strategies are determined from this perspective. To rank from this word what can we do? We can use the word and related words in our content repeatedly. That way we can sign to Google or other search engines that our site is related to this word. From that part, search engines check this issue and crawl the website. While crawling a website, if a search engine sees such content in the website, crawl bots can inform the main machine and the main machine could decide to get this signal to make the website’s page rank higher with this word. While doing this, many websites will accomplish such logic to rank with these keywords. Thus, what other metrics could Google and other search engines include in their calculations? Let’s explain this in the next context.
Especially Google has a variety of algorithms to check page rank at its ranking pages with keywords. These algorithms are Fred, Hummingbird, Intrusive Interstitials Update, Pigeon, Mobilegeddon, Payday, RankBrain, Exact Match Domain, Panda, Page Layout Algorithm, Penguin as they could be seen. If we could explain these algorithms shortly; the Fred algorithm could be viewed as tracking websites just monetizing the website and not solving the user’s problem. From that perspective, Fred updates(a bunch of algorithms) try to elect black hat SEO tactics and degraded user value on the site. For example content is low but traffic is high with unrelated ads and so on. Fred degrades this site rank as well. Hummingbird is an algorithm change made in 2013 by Google. Hummingbird algorithm seeks relations with web pages and landing page direction by Natural Language relative queries. If the directions and link properties and anchor text are relative and supportive for user direction, the Hummingbird algorithm could reward your web pages. That means, in a 4-5 or more linking to sell button or something to game or app, you should create a good content path on the way. That way, it is checked by Google with Hummingbird Algorithm. If your ads and other content at the top of the page and block users to see content and other links to visit web pages, then Intrusive Interstitials Update could harm your website's potential at ranking. This update looks at bounce and relative subjects that blocks website function by mainly controlling ad structures and popups on the website. Pigeon algorithms seek local results and consider distance and location. Therefore, the local related and distance related pages to user query will be higher in the SERP(Search engine result page). Mobilegeddon algorithm probably seeks signals from the difference of traffic metrics from desktop and mobile. From that perspective, if the difference is huge and mobile traffic is greater, this algorithm could reward your SERP rank on mobile searches. It is not included in desktop searches. It could also get statistical data of differences between desktop and mobile signals and reward the healthful disparity to general. Payday algorithm’s main purpose is to penalize the website that uses spam content techniques to rank. For example the spam keywords without related content and other usages of links, properties could be included in this topic. RankBrain algorithm searches entities related to Hummingbird natural language processing. By doing that, it specifies true entities from that webpage and marks them. Exact Match Domain refreshes web pages that spam keywords while not writing content with these specific keywords. By doing so, pages could have ranked higher before. With that update, you cannot just write specific keywords for matching on the page. Panda algorithm controls the content and its quality about many factors. These factors generally analyze the page content by originality, insight, and quality. That way thin and spammy contents could not rank high. Page Layout Algorithm checks the site ads at the top of the webpage. If these ads are so many and block the content from reading, then this algorithm punishes the web page for doing so. Penguin algorithm checks the backlink status, trust flow, and domain authority. That way, spammy and unrelated backlinks become a fact for penalizing site rankings.
Search Engine Optimization Ranking Factors
Domain Age
Domain age is important and Google and other search engines could behave differently to old and new websites. Therefore, aging could be a benefit for your website.
Keyword in Domain
Using keywords on domain name could give you a boost but it is not valid factor today yet it is a factor as seen.
Domain Registiration
Domain registiration period is a small factor also for ranking. Valid domains refresh contionously.
Keyword in Subdomain
The keywords are included in subdomain is a again a factor that could rank you some specific.
Domain History
The panelized domain name or its punished usage could pass to new user domain
Exact Match Domain
The query domain name matching tactic could give you some boost but it is degraded by this named algorithm as it is mentioned.
Public or Private WhoIs
Private whoIs could signal that owner may have several websites also which is faulty for your leading website domain
Public or Private WhoIs
Private whoIs could signal that owner may have several websites also which is faulty for your leading website domain
There are many factors as we can see they can sum up to huge pail of story. However, we can analyze the Semrush errors and titles they give for websites. You can look further these articles here google ranking factors
Semrush Factors About Their Technical SEO Calculation
There are three parts of a SEO audit. Technical SEO, Off-page SEO and On-page SEO. Technical seo is about html, tag configuration and css, javascript applications for mobile and destkop version of website. Therefore technical seo is about technical details of website configuration for better user experience. On-page seo could be summarized as content structure and Off-page seo could be summarized as backlink strategy. We can see Semrush technical factors below.
Structured Data Items and their building on web pages
Pages that is incorrectly stated in sitemap and should state the web links correcty
Title, meta description, content duplicate errors and their reorganizaiton
Orginal title tag usage in every, canonically orginally stated web pages
3 second below web page open time
low text-html ratio problem
Duplicate h1 and title problem that they should be unique
Cache necessity of the static files
h1 heading necessity for every orginal canonically stated pages
too short or too long title tag texts
Anchor text could be needed for links
404 and broken links on web pages
Not crawlable web pages
Not stated or written alt tag decsription for images
Minification of Javascript and CSS files
301 and 302 redirect errors
Correct URL formation using “-“ instead of “_” and further
Broken 404 error image folders to sources
Orphan urls in sitemap
These are the main issues that Semrush checks for websites when they crawl the URLs with their bots. These issues could be repetitive for many web pages. When we complete these issues, we can say that we have been in a state that we had completed half of the technical SEO. However, there is a factor Google generated recently which is named “Core Web Vitals”. These performance metrics oriented subjects will permanently affect the web pages after May 2021. Therefore, these will be valid issues after May 2021. I will explain the on-page, off-page SEO, and core web vitals terms in the next paragraphs.
On-Page Search Engine Optimization Factors and Content Related Matters
There are several ranking factors on a webpage about on-page matters. Keywords here are an important entity as we described before. In here, this entity meaning is another thing. It is the keywords functions as signals for SERP. That’s why using major notice for google algorithms could be so beneficial. Because we have several choices to use these keywords and logically if we use them at important tags, crawl bots will see these and mark this website with related natural entity-body. Therefore using keywords in title, meta description, and h1 tag is crucial. Slightly spamming keywords on p tag as a text was a great signal factor in the past and still, it has some power. Informative and academically sounding articles getting trendy and that’s why these could be great for users. Because every website has some similar information and the user wants to see original and real content. Another subject is a similar content issue as we saw at Semrush error types. This issue could be solved by canonical tag. When we give a more main page this canonical tag, this expansive scope includes the whole pages and the crawler only parses the canonical given page. That’s why a similar copy content issue could be solved and the update news mentions that this issue is not a problem anymore. Therefore, SEO tools could be updated soon about this subject. So we write content that is new, authoritative and seems an expert content yet after a while the information and details about the topic could change and directs a reverse path. That’s why we should mention new details and elements about our subject and update our content when it is necessary. Another undervalued issue is giving backing or direction to another page that has great authority and great content with high traffic. With the true intention, Google and other search bots could reward this act. Interlink configuration is also important about content and subject and its flow of connection. And using keywords in URL or using URL related words as in content is a specific factor for ranking.
Off-Page Search Engine Optimization Factors and Content Related Matters
There are lots of factors for off-page SEO. However, link building is one of the main factors on all of them. Because it works as promotion of a site from the other web site. It suggests this website on its source as a friend or person suggest another person for a job and so on. Therefore, external link buildings from authoritative sites and related text inside anchor tags could affect majorly a website. Google sees this as a great factor for ranking so it rewards websites that getting links from other sites. Social media and og tags are also part of the ranking system and they signal ranking and traffic. Influencer marketing could be counted for off-site SEO strategy. Being online and posting regularly on online sources could make a great factor after a time. There are some factors for important backlink configuration. The popularity of a webpage getting a backlink is a critical factor. Another thing is related content of the website that do they match. Other factors are trusted, freshness, and anchor tag usage. These factors will reward ranking if they are stated nicely.
Core Web Vitals and Web Site Performance Metrics
That is stated that in 2021 May, the core web vitals will be a huge factor for rankings officially. Therefore, these factors should be taken seriously. What are those web factors? These are site speed & performance factors measuring user activity, web site reaction, and more. We can summarize these vital with three main metrics. These are the Largest Contentful Paint, First Input Delay, and Cumulative Layout Shift. The largest Contentful pain is the render time of the largest text or image visible in the layout. First Input Delay is the page viewed and interactivity time period. When javascript is ready to react and event handlers are ready, this measurement is finished to be taken. Cumulative Layout Shift is the product of shit ratio and shifts impact ratio. The sum of these gives the total score. There are other metrics that the lighthouse includes in its measurements. For example, First Contentful Paint is a measurement of the first part that is shown when the website renders. Time to Interactive is the time passing from FCP to all javascript activities are finished. Total Blocking time is the sum of javascript event that passes 50ms time period. General problems about these measurements are: removing unused javascript and CSS files, new image format transformation, correct image resizing, http/2 usage over http/1.1, connecting third party resources, preloading static files, caching static files and compressing these files, and minifying javascript and css’s. When you solve these problems that lighthouse specifics, site performance increases, and you get a higher level of the score which rewards you at SERP.
So as we can see, search engine optimization has many issues and these could be detailly complex. However, completion of these matters may make your website's organic traffic high and you could succeed at your website monetization.
