My Hobby of Algo Trading
The emotional and time-consuming hobby of trying to eliminate emotion and time consumption from trading
What is Algorithmic Trading?
I jokingly refer to Algo Trading as the emotional and time-consuming hobby of trying to eliminate emotion and time consumption from trading. Like many, I was inspired to get started because I wanted to reap the benefits of successful trading without the time commitment and emotional ride. However, I quickly learned that it is not that simple and wound up spending an equal amount of time trying to make trades automatically as I had manually trading.
Simply put, algo trading is the development of virtual robots that replace a trader. These trading bot programs are completely autonomous but follow a set of defined rules (some looser than others). Part of successful autonomy is the ability of the program to make decisions in real-time as underlying market conditions change.
What does it look like?
When most people visualize the stock market, they imagine either an open-call trading floor with people yelling and bidding or they imagine a hedge-fund setup with monitors full of charts. Both of these examples are useful ways for humans to visualize and interpret data, but computers don’t need visualization; they only need the data and instructions on how to interpret it. When someone develops a trading algorithm they are usually trying to get the computer to see the market the way that they see it. If the goal is to directly transfer a manual trading strategy into a autonomous strategy then all you need are three sets of criteria: Entry, Holding, and Exit.
This TSLA daily chart shows trade Entry criteria in yellow. TSLA changed from overbought in December 2022 to oversold in January 2023 with Bollinger Bands and the Relative Strength Index (RSI) giving clear indicators that a bullish play was being set up. A simple trading bot would have a list of if/then statements that form the trade entry criteria and when all boxes are checked within a given time period the bot would execute a trade entry.
In green we see an example of Holding criteria. This can be continuously evaluated by the bot as the trade is held. The stock continues to rise while not decreasing by more than the Average True Range (ATR) recorded at the time of trade entry and the RSI remains safely above it’s moving average.
The blue circles indicate trade exit criteria. The stock has three bearish daily candles in a row which was preceded with a sharp drop in both ATR and RSI. While this criteria doesn’t guarantee the absolute top (and you’ll never predict that no matter how hard you try) it does give a strong signal to exit the trade and results in a sizeable profit. While this is an oversimplified and cherry picked example, it demonstrates how a program can evaluate data in a similar fashion to how traders visualize it.
How I started
I had essentially zero knowledge of the stock market until 2018 other than having met a few traders in passing and watching movies like The Wolf of Wall Street. I didn’t imagine that I could successfully participate until I was deployed and saw some of my colleagues making successful investments. They were talking about it every day and seemed to be making some good money on the side. I thought to myself “if these guys can make money doing this then surely I can figure it out too”.
With my curiosity piqued, I began reading and researching. The first book that I read was Stock Investing for dummies which I highly recommend to anyone with no prior knowledge. It does a great job of breaking down the fundamentals and is pretty easy to digest. I also found a few YouTube channels that introduced me to technical trading and swing trading. Of course these days YouTube is oversaturated with terrible get rich quick clickbait and lots of trading course promotion, but in 2018 there weren’t nearly as many people making trading content.
After several months of reading and learning, I started setting goals of trading manually in 2019. I started paper trading on Trading View for a few months until I felt comfortable risking real money and started to develop some amateur strategies. After some moderate success with delayed market data, I opened up my first brokerage account and started buying stocks for swing trading plays.
Shortly after, one of my friends asked me “Have you ever tried trading options?”. I was doomed. One of the worst things you can do as an amateur trader is jump into high risk assets like options contracts. I learned some hard lessons in 2019, but was nevertheless determined to continue. I was fortunate to find some mentors that worked in the same building as me and had decades of experience. When work was slow we would go over charts and strategies together and I learned how to take a more reasonable and realistic approach to investing.
Over time we actually came up with some successful strategies and were having a great time making some extra money on the side. I was absolutely hooked! I’ll never forget the rush of a market-open set up. My heart would be pounding and palms sweating while I anxiously watched the premarket chart waiting for the bell. Sometimes it paid off tremendously and other times I took a quick loss and spent the next several days going over what went wrong and why.
The only issue was that I was spending all of my free time on this hobby and it was getting difficult to manage. I was fortunate to be living in Europe so I could work my day job and then trade the US Market in the evening, but after a while it felt like I was working two jobs and I was quickly losing sleep and patience. My weekends were consumed with charts and excel spreadsheets where I would track 50-60 stocks at a time and mark tickers that would have potential set-ups in the following week.
At the end of 2020 I thought to myself “There has got to be a way to automate this”. I had gotten part of the way there with some excel formulas, but I was still chained to my computer while I had open positions. With no prior coding or software experience I decided to learn how to build trading bots.
I started with Trading View’s Pine Script which isn’t really a language but was a useful starting point. I quickly figured out how to code some simple strategies and backtest them. The results seemed great! Of course they were actually too good to be true, but I was nevertheless encouraged to keep going.
My next step was learning Python. I had heard that it was a very popular language and that it was relatively easy to learn compared to C# and others. After some research I found the QuantConnect platform which allows you to code trading bots in Python and deploy them live from one platform. Instead of taking a reasonable approach and learning the fundamentals of coding I dove in head first and started figuring out how to code a trading bot.
I essentially brute force attacked my way into building my first program. It was lines upon lines of if/then statements and took months to get running without errors. But eventually I had a program that could perform live autonomously. It wasn’t very successful and I learned a lot of hard lessons, but it was something that could operate on its own and I was very proud of it.
My Approach
Over the last 3 years I have spent most of my free time writing programs and deploying them to trade both stocks and options. Through trial and error I have developed an approach that for me makes sense and has lead to immense learning as well profitable algorithmic trading.
When coming up with a strategy or model, I always start with an observation or a question. In my early days of development I would try to just plug in the idea and then spend hours backtesting and overfitting with QuantConnect’s LEAN engine. Eventually I realized that I was spinning in circles with no real basis of why I was programming what or why I was getting certain results. I changed my approach to be based around thorough research, not just hunches.
Using a Research First approach allows you to make determinations on whether an idea or strategy is effective or even worth pursuing. I started using Jupyter notebooks and NumPy arrays to evaluate data and find the why. This proved to be much faster than running backtests in LEAN over and over again and also allowed me to think more thoroughly about the best way to implement strategies.
Research can provide SMART objectives: Specific, Measurable, Attainable, Realistic, and Time-bound. How can you expect to deploy a model successfully if you have no idea how long a position should be held, what the average expected return should be, or what the downside risk is? If you a developing with an unrealistic goal of making 1000% on your trades then you are fated to fail.
I believe in Macro sets and Micro spikes. You wouldn’t try to spike a volleyball that was already falling below your waist. In the same respect, your strategy should be set up by the current macroeconomic environment. If a given sector is on a downtrend, it probably isn’t the best time to deploy a bullish strategy.
I rely heavily on Probability and Statistics. If you haven’t already guessed, I am a technical trader. I’ve always been about the numbers and not so much fundamentals. This isn’t the only approach but it is what works for me. I trade based on what I can calculate will be the highest probability of positive return and sustainability.
I build as much as I can from the ground up including programming my own indicators. If you have ever looked at a chart from Yahoo Finance, Trading View, and Interactive Brokers at the same time you probably noticed that their indicators are all slightly different. This is because every platform and brokerage uses their own smoothing algorithm. None of them are the same and all of them skew from the mathematical formula that the indicator was derived from. Coding your own indicators rewards you with the bonus of forcing you to fully understand what you are using and how it is calculated. It also provides you with consistency and accuracy.
I also created my own order tracking system and a redundant data backup system. When deploying an algo live, I ran into the issue that if the server got kicked offline or the program lost it’s connection with the brokerage all of my portfolio information would be lost. I might still have 5 open positions but I couldn’t see or close any of them without manually logging into the brokerage. Creating my own redundant system mitigates that risk and ensure that all positions will be recorded and tracked accurately.
I only trade highly liquid assets with at least 1-10m daily volume which helps to ensure order fills. I’ll never forget learning this lesson the hard way in 2019. I had done my analysis and made what seemed like a great play on a stock that was moving up. I had bought OTM call options on the underlying asset and when the stock price surpassed my strike, my account was a showing a sizeable return. However when I went to close the position my order would not go through. I angrily called my brokerage complaining that my order wasn’t being filled. The response I received was “well there has to be someone on the other end willing to buy your position for that price”. There wasn’t.
I develop different models for different underlying market conditions. This falls within the Macro sets, Micro spikes tenant but I believe it deserves some more emphasis. Many people try to make one algo for all market conditions. This just doesn’t exist. You are much more likely to succeed if you have a collection of models that can be deployed and adapted as market conditions change.
Common Mistakes
For as many mistakes as you can make with manual trading there are an exponential number of mistakes that you can make attempting to algo trade. Here are some of the big ones that I have either experienced personally or see others make very often.
The absolute #1 worst and most common mistake that even experienced algo traders make is Overfitting. If you find yourself finely tuning your parameters and these tweaks are the difference between a positive return and a negative return: STOP. You are overfit. Overfitting is the adjustment of parameters to capture outliers and increase the score of your testing phase.
This diagram gives a visual representation of how a classification model can be overfit. You can see that the optimal fitting isn’t perfect, but it is correct most of the time. The overfit model attempts to capture all of the outliers and get a perfect score. This isn’t realistic. The stock market is dynamic and is subject to variables like major world events, politics, scandals, etc. You will never capture all of the outliers and the outliers will never be exactly the same. Accept that every strategy has inherent risk and if you’re strategy’s success relies on hitting every win and mitigating every loss, then it isn’t worth pursuing.
Another common mistake, especially with High Frequency Trading (HFT), is Underestimating Slippage. Slippage is the price change between the time an order is placed and the time the order is filled. One of the treacheries of backtests is that they assume your order will be filled instantly and at exactly the right price. This is very unrealistic. When slippage is overlooked you run the risk of losing money on every trade despite the testing data showing that you should be making money on every trade. HFT isn’t my style, but I have seen others make this mistake many times.
A mistake that I learned from the hard way is Using Unverified Data. Unfortunately even when you pay good money for data, it can be incorrect or incomplete. I used to assume that all of my data would be complete and accurate until I started running into problems. When I tested over certain dates I would start getting errors and my program would fail. I spent hours trying to figure out what was wrong with my code only to eventually realize that some of the data I was using was either mislabeled or missing big chunks.
An easy method to help mitigate this is to visualize your data by plotting it and visually checking that it looks correct. This won’t work 100% of the time but it is usually easy to see when something is missing or seems off. When you are using a very large dataset with hundreds of thousands or millions of datapoints this gets trickier. But one solution I’ve used is to randomly sample sections of the data to visualize it and when I run into errors I always check the data at the point of error before assuming that it is a bug in my code.
Another common mistake is Over Complicating Strategies. Usually this involves adding a large number of parameters and trying to account for any and all possible outcomes, in other words: overfitting. Making your algo too complicated also has the side effect of the program being very large which makes it harder to debug. On the other side of the same coin Over Simplifying Strategies can be equally egregious. I promise you there is no such thing as a magic indicator or a perfect strategy. If your strategy hinges on a single indicator or parameter then it most likely won’t work in a live dynamic environment.
Whether you are training a machine learning model or backtesting, do not make the mistake of Failing to Test Out of Sample. You should always have a testing dataset and a validation dataset. Sometimes when you are trying to capitalize on specific market behavior you may run into the issue of lack of available data. There are usually other stocks or ETFs that have somewhat correlated moves that you can use. There are also methods for augmenting data that allows you to increase the size of your dataset. Otherwise you increase your risk of overfitting as well as failing to realize potential pitfalls of your strategy.
Lastly another common mistake is Failing to Implement Risk Management. It doesn’t matter how good your strategy/bot performs in your tests. If you don’t develop a risk management system then you are opening yourself up to huge potential losses. The simplest risk management scheme is a maximum drawdown or maximum number of losing trades in a given period. If your bot performs negatively outside of your projected downside risk or takes repeated losses, an emergency stop could save your portfolio.
Model Methods Used
Each of these methods deserves their own article but I will quickly give an overview of some of the methods I have used.
I love this cartoon because it pretty much sums up Machine Learning. We don’t know what calculations are happening inside of the black box, we just know when it gets the answers right. I started learning how to build and train neural networks a few years ago because I felt that AI would be the last word in trading. The most important thing to know about machine learning is that the bulk of the work in in preparing the data to feed into the model. You can create a neural network in just a few lines of code in Python thanks to packages like SciKitLearn or Tensorflow. However if your data isn’t organized and labeled (for supervised learning) correctly, then the model will not give you the right answers. If you want to learn the basics of machine learning and how to build a simple neural networks, I highly recommend this course from SoloLearn.com. It is what I used when I was getting started and it gave me a great jumping off point.
I have used Linear Regression not as a strategy but as a tool for optimization of trade timing. Sometimes your trade entry criteria is being met but the stock price is still on it’s way down. You can use linear regression as a tool to gauge when the optimal time is to take a position as well as evaluate if a trade should be made at all.
Autoregressive Models take the previous value of a variable, the current value of a variable, add some randomness and predict the next value. This is a gross oversimplification but that’s the basic idea. I’ve found this method to be great for data augmentation when I want to create more testing or training data.
Markov Chain Monte Carlo is one of the hardest and most interesting models that I have utilized. There are many branches of MCMC but I found success using the Metropolis Hastings algorithm. I’ve used MCMC for both data augmentation and price prediction. For me it was quite a learning curve but once you get the hang of it the results can be very rewarding. I also enjoyed the underlying philosophy.
Most recently I developed a model that uses Variable Candidate Selection. You can learn more about this method in my research paper posted Here on Substack. The short version is that I wrote a program that evaluates 200 highly liquid stocks every day and evaluates their recent history and correlating strategy indicators. It then filters them into groups and selects the group with the highest probability of success as potential candidates for that day’s trades.
If you would like to see a more in depth look at one of these methods or anything else discussed in this article, please leave a comment and let me know! I hope that you enjoyed reading this as much as I enjoyed writing it and reflecting on my journey.