Freqtrade Stuff

Where To Start With Freqtrade

2021-04-20T00:00:00+00:00

Hi, I’m new. Do you have any advice for beginners?

I am not an expert, and nothing that follows is financial advice. These are my opinions based on my experience so far.

Financial Advice

Trading is risky, and trading crypto is especially risky. Only trade what you are willing to lose.

My first priority is preserving my capital.

Crypto trading is not for you. You may want to look into a balanced ETF portfolio, bonds, GICs, or a high rate savings account. Speak to a qualified financial advisor.

I want to make crazy profits, and I’m willing to risk losing everything I’ve invested for the chance to get them.

Carry on.

Basics

There are broadly 3 different areas you should become familiar with in order to trade using Freqtrade.

General trading terminology, and technical analysis.
A baseline of technical skill, and basic programming knowledge. The ability to read and understand Python code.
How to use Freqtrade specifically, its features and limitations.

General Trading

You should have a basic understanding of how trading works in general. Be sure to understand the terms market order and limit order and the difference between them, bid-ask spread, volume, slippage, how the order book works, spot vs. margin trading, how a stop-loss order works.

All of this information is common to classical trading and crypto trading, search for them on Google or Investopedia.

Once you’ve got the basics of trading covered, you’ll want to dig deeper into Technical Analysis, which involves looking at historical price data and determining patterns which you expect to repeat in the future with enough confidence to inform your trading decisions. Technical Analysis typically uses chart patterns or technical indicators.

This is in contrast to Fundamental Analysis which focuses on looking at the actual performance or value of companies or assets which are being traded, news stories, public sentiment, etc..

Freqtrade strategies typically focus entirely on Technical Analysis.

Guide To Technical Analysis

Familiarize yourself with basic indicators such as moving averages, MACD, Bollinger Bands, and RSI.

If your exchange supports paper-trading (simulated, with no real money), then choose a crypto currency to practice trading with. Set up a chart on your exchange or TradingView that has a few indicators that interest you, and make some paper trades based on them.

Be sure to read not just what the indicators show directly, but about how they are typically used by traders, and for what purpose. Some monitor trends, some can be used as signals to make a buy or sell, etc…

How to Combine Trading Indicators

Programming Skills

I don’t recommend using Freqtrade without at least basic understanding of programming fundamentals and the Python language.

Freqtrade does not come with a “default” strategy that makes money. You must evaluate each strategy that you intend to use to make sure that at a minimum it does something even remotely sensible, and in the worst case is at least not actively malicious.

If you are unable to at least review a strategy in order to spot obvious mistakes or potentially malicious code, then you will be at the mercy of whoever provided the strategy to you.

At a more general level, strategies will perform differently depending on the exchange they are run on, timing, the current market conditions, the trading pairs used, the timeframe used, the specific configuration of your freqtrade bot, and so on. You may need to make at least basic changes to the strategy code in order to make it more suitable your specific situation.

Python For Beginners

If you aren’t comfortable or interested in learning Python basics, then Freqtrade may not be the best fit for you.

Freqtrade Specifics

A few key points:

Freqtrade does not automatically make money. There is no “default” strategy that you can turn on to earn a profit. Strategies are written in Python by you, or shared by other users.
Making profit trading is challenging, time consuming, and not at all guaranteed. The same goes for automated trading, although ideally it becomes less time consuming when (and if) you find a strategy and configuration that is working for you long term and can be run with minimal active management.

Freqtrade supports spot trading only. You cannot use it to make short trades, to trade using margin, or to trade options contracts or futures.

Freqtrade does not support position stacking. You cannot use Freqtrade to gradually buy more of a coin. Once a trade is opened, it has to be sold before another trade can be made for the same coin.

Getting Started Flow

Install Freqtrade using the instructions on the official website, either using Docker, or directly on your machine.
Grab the free strategies provided in the separate freqtrade-strategies github repository. Some of the strategies collected under user_data/strategies/berlinguyinca/ in particular are excellent starting points or references for developing your own strategies, and will often be mentioned by other Freqtrade users.
Download the data that you’ll need to use for backtesting using the download-data command.
Refer to the Configuration documentation, as it will be necessary to make changes to your configuration in order to get much of anything else done.
Start backtesting strategies using the backtesting command.
Once you’ve run a backtest or two, use the Plotting functionality to visualize the results of your backtests. This can be incredibly helpful in understanding what your strategy is actually doing. Try out both the plot-dataframe and plot-profit commands.
Try out developing your own strategy.
Before putting real money on the line, or even setting up your exchange keys, choose a strategy to run in dry-run mode, which will simulate trades in realtime using public data provided by your exchange.
Set up Telegram to monitor the bot during dry-runs.

Make use of the search function at the top of the Freqtrade Offical Website first, whenever you have a question about a specific command, config parameter, or topic. If you don’t find what you’re looking for, you can ask for help on Discord or Slack.

Backtesting Traps

2021-04-12T00:00:00+00:00

Backtesting is important to understand the performance and behaviour of your strategy, but there are a number of issues which can result in a backtest producing much better results than are likely once the strategy is run live (or dry-run).

This information applies to the Hyperopt feature as well, which processes strategies in the same way as Backtesting.

To start with, make sure to familiarize yourself with the official documentation on Backtesting and it’s assumptions.

Basics

The Future Will Be Different From The Past

Backtesting tests use historical data, so even if your strategy avoids all of the other problems, there’s no guarantee that it will succeed in the future, even if it succeeded on past data.
Trade Timeouts

When backtesting, all of your trade orders are magically filled at the exact price you wanted. This doesn’t happen in real life, and if you place limit orders, it’s likely that some percentage of them go unfilled and time out, causing you to miss entering or exiting a trade. The likelyhood of this happening depends on the volume of the pair you’re trading, when your strategy places orders, whether the market is moving up or down, which side of the bid/ask spread you place your orders on, etc…
Slippage

Regardless of which order type you use, the market will continue to move between the time your strategy decides to place and order, and that order actually getting placed, and hopefully filled on the exchange. Particularly when using market orders, this causes price slippage and it is likely that you will get a somewhat worse price than the moment the strategy decided to place an order. How much worse will depend on the exact market conditions, spread, volume, the size of your order, price direction, etc…

Refer to Prices Used For Orders and Market Order Pricing in the Configuration documentation for more details.

Things to watch out for when backtesting:

Large number of trades and a low average profit. If your strategy is profitable in backtesting based on many trades with very small profits, it’s very likely those profits will become losses in live due to slippage, especially on low volume pairs. Be cautious if your average profit is below 0.5%, and wary of any changes that increase your total profit but decrease your average profit.

Reading Future Candles

Something that can be difficult to get used to when backtesting is that signals are generated from the entire time range at one time. The bot does not start at the first candle and process each candle in order.

populate_indicators, populate_buy_trend, and populate_sell_trend are called once for a given pair, and generate all of the signals for the entire duration. This means that the “end” of the Dataframe is always the final candle of the backtesting time range, and is not the “most recent” candle as it would be when running live or dry.

When creating the signals for any given candle it is therefor easy to accidentally read from the candles which follow in the Dataframe, effectively “seeing into the future”. When running dry or live, the future is not available.

Read the official documentation on common mistakes made when developing strategies for details on things to avoid.

Trailing Stops

“i keep tightening trailing stoploss and backtest profits go up. that cant be right.”

One backtesting anomaly that seems to produce incredible, one might even say unbelieveable, results is the use of very small trailing stops.

Here’s an example strategy that exploits this feature. It uses no signals, and simply buys on every candle on a 1h timeframe. Sells happen by reaching minimum ROI of 1%, or by hitting the trailing stoploss of 0.1%.

from freqtrade.strategy.interface import IStrategy
from typing import Dict, List
from pandas import DataFrame

class Magic_Trailing_Stoploss(IStrategy):

    minimal_roi = { "0": 0.01 }

    stoploss = -0.01
    
    trailing_stop = True
    trailing_stop_positive = 0.001

    timeframe = '1h'

    def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        return dataframe

    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        dataframe['buy'] = 1
        return dataframe

    def populate_sell_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        dataframe['sell'] = 0
        return dataframe

Now we backtest…

In only 11 days, we’ve gone from $2,500 to $65,150, a 2506% profit! Amazing!

Sadly, this strategy will not produce similar results when run live.

First of all, take a look at the Avg. Duration Winners / Losers. 1 minute, and 7 minutes respectively. When trading hour long candles, this is a sign that you’re exploiting the behaviour of backtesting.

Using plot-dataframe can further help illustrate what’s happening:

Most of the candles show that the trade opened and close in the same candle, which backtesting will consider to be 0 minutes, or on the very next candle.

What’s Going On?

When backtesting at any timeframe, Freqtrade has only candle data to work with. It doesn’t know how the price moved during the candle. With 1m or 5m candles, it’s less of a problem (although it still exists), but with longer timeframes, like 1h, 4h, etc… Freqtrade needs to make some major assumptions about the order of events.

As mentioned in the official documentation on Backtesting, when Freqtrade calculates trailing stoploss, it first moves the price to the high, moves the trailing stop up to match accordingly, and then moves the price down, possibly triggering the new trailing stop price.

Backtesting with extremely tight trailing stops essentially gives you a perfect candle trade, selling just below the high of the candle, almost every time.

The longer the timeframe you use, the less realistic this is. With a small trailing stoploss, it’s exceptionally unlikely that the price would move directly from open to high without decreasing enough to trigger the stoploss. If your stoploss is smaller than the spread on the pair you’re trading, the stoploss will likely be hit on the next sell trade. The longer the timeframe, the more likely your trailing stop, or even your default stoploss, will be triggered before reaching the high and then falling.

ROI

If you have a reasonably tight ROI set, you may see the ROI hit instead of the candle high, but the same trailing stop issue applies. The longer the candle, the less likely the price will actually reach the ROI before triggering your trailing stop.

What To Watch Out For

Things to watch out for while you’re developing/backtesting your strategy that might be a clue that you’re accidentally exploiting this behaviour:

Average duration of trades is lower than your candle timeframe, maybe only a few minutes when using 1h candles.
When plotted, trades regularly close on the same candle that they are opened.
Your trailing stop is lower than, or not much bigger than the spread on the pairs you’re trading. Typical spreads range from 0.1% to 0.5% on high volume coins, and are often much higher on low volume coins.
Profits keep increasing as you decrease the size of your trailing stop.

Trailing Stop Workaround

One solution you can use to ensure that you are seeing more realistic backtest results is to backtest your 1h strategy at 5m or 1m, using an informative pair at 1h to generate trade signals. The signals will operate on the same 1h timeframe that live does, and running the strategy using 5m or 1m candle data will simulate price movement during that 1h, providing more realistic trailing stoploss and ROI behaviour.

Unfortunately this will be slower than backtesting at 1h, and it complicates the code somewhat. But if you plan on using a trailing stoploss or ROI, you probably want to know that your backtest results are not complete lies.

Here is a simple strategy that uses a simple EMA crossover to generate buy signals:

EMA_Trailing_Stoploss

from freqtrade.strategy import IStrategy
from typing import Dict, List
from pandas import DataFrame
import talib.abstract as ta
import freqtrade.vendor.qtpylib.indicators as qtpylib

class EMA_Trailing_Stoploss(IStrategy):

    minimal_roi = { "0": 0.01 }

    stoploss = -0.01
    
    trailing_stop = True
    trailing_stop_positive = 0.001

    timeframe = '1h'

    def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:

        dataframe['ema3'] = ta.EMA(dataframe, timeperiod=3)
        dataframe['ema5'] = ta.EMA(dataframe, timeperiod=5)
        dataframe['go_long'] = qtpylib.crossed_above(dataframe['ema3'], dataframe['ema5']).astype('int')

        return dataframe

    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        dataframe.loc[
            qtpylib.crossed_above(dataframe['go_long'], 0)
        ,
        'buy'] = 1

        return dataframe

    def populate_sell_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        dataframe['sell'] = 0
        return dataframe

Backtesting shows a very respectable 11% profit, but this is an illusion:

And the same strategy after converting it to be backtested at 5m or 1m:

EMA_Trailing_Stoploss_LessMagic
change timeframe to 5m and add informative_timeframe
move indicators from populate_indicators to a new function called do_indicators
populate_indicators checks timeframe and run mode, and runs do_indicators directly when live, or runs it using the 1h informative pairs and then merges the results back into the main dataframe when backtesting.
put all of your signal creation in do_indicators, and only read buy/sell signals from populate_buy_trend / populate_sell_trend

from freqtrade.strategy import IStrategy, merge_informative_pair
from typing import Dict, List
from pandas import DataFrame
import talib.abstract as ta
import freqtrade.vendor.qtpylib.indicators as qtpylib
from freqtrade.exchange import timeframe_to_minutes

class EMA_Trailing_Stoploss_LessMagic(IStrategy):

    minimal_roi = { "0": 0.01 }

    stoploss = -0.01
    
    trailing_stop = True
    trailing_stop_positive = 0.001

    timeframe = '5m'
    informative_timeframe = '1h'

    def informative_pairs(self):
        pairs = self.dp.current_whitelist()
        informative_pairs = [(pair, self.informative_timeframe) for pair in pairs]
        return informative_pairs

    def do_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        dataframe['ema3'] = ta.EMA(dataframe, timeperiod=3)
        dataframe['ema5'] = ta.EMA(dataframe, timeperiod=5)
        dataframe['go_long'] = qtpylib.crossed_above(dataframe['ema3'], dataframe['ema5']).astype('int')
        return dataframe

    def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:

        if self.config['runmode'].value in ('backtest', 'hyperopt'):
            assert (timeframe_to_minutes(self.timeframe) <= 5), "Backtest this strategy in 5m or 1m timeframe."

        if self.timeframe == self.informative_timeframe:
            dataframe = self.do_indicators(dataframe, metadata)
        else:
            if not self.dp:
                return dataframe

            informative = self.dp.get_pair_dataframe(pair=metadata['pair'], timeframe=self.informative_timeframe)

            informative = self.do_indicators(informative.copy(), metadata)

            dataframe = merge_informative_pair(dataframe, informative, self.timeframe, self.informative_timeframe, ffill=True)
            skip_columns = [(s + "_" + self.informative_timeframe) for s in ['date', 'open', 'high', 'low', 'close', 'volume']]
            dataframe.rename(columns=lambda s: s.replace("_{}".format(self.informative_timeframe), "") if (not s in skip_columns) else s, inplace=True)

        return dataframe

    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        dataframe.loc[
            qtpylib.crossed_above(dataframe['go_long'], 0)
        ,
        'buy'] = 1

        return dataframe

    def populate_sell_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        dataframe['sell'] = 0
        return dataframe

Backtesting now shows all of the profits have vanished, this is much closer to what would happen live, and will be even lower once spread, slippage, unfilled orders, etc… are accounted for.

Why TheForce Strategy Doesn’t Work

2021-03-29T00:00:00+00:00

TheForce gets amazing backtest results, but doesn’t work live.

The crazy backtest profits are only seen when using signalperiod=1 with ta.MACD(dataframe,12,26,1)

If we switch the MACD implementation from ta-lib, to qtpylib, most of the profits go away:

        # MACD (Magic Profits)
        macd = ta.MACD(dataframe,12,26,1)
        dataframe['macd'] = macd['macd']
        dataframe['macdsignal'] = macd['macdsignal']
        dataframe['macdhist'] = macd['macdhist']

        # MACD (No Magic Profts)
        macd = qtpylib.macd(dataframe['close'],12,26,1)
        dataframe['macd'] = macd['macd']
        dataframe['macdsignal'] = macd['signal']
        dataframe['macdhist'] = macd['histogram']

================================================================ STRATEGY SUMMARY ===============================================================
|        Strategy |   Buys |   Avg Profit % |   Cum Profit % |   Tot Profit USD |   Tot Profit % |   Avg Duration |   Wins |   Draws |   Losses |
|-----------------+--------+----------------+----------------+------------------+----------------+----------------+--------+---------+----------|
|      TheForceV7 |   1872 |           0.88 |        1641.61 |        24501.162 |        2450.12 |        0:56:00 |   1297 |       0 |      575 |
| TheForceV7_test |   1853 |           0.11 |         209.05 |          506.040 |          50.60 |        0:32:00 |    706 |       0 |     1147 |
=================================================================================================================================================

Why?

What seems to be happening is that ta-lib is, for some reason, cheating by looking into the future by 2 candles when signalperiod=1.

This is an overlay of each implementation using signalperiod=2, the results quickly converge and overlap each other.

This is an overlay of each implementation using signalperiod=1, the ta-lib version is looking into the future by two candles. You can see that the ta-lib signal line is two candles “early”, predicting the future price movement.

Just double checking, here is each implementation using signalperiod=1, but the ta-lib version has been shifted forward by two candles using .shift(2). The macd line is now two candles “late”, and the signal line once again lines up with the qtpylib implementation.

Sadly TheForce is looking into the future to predict the direction of price movement, which can’t happen during live or dry-run, unless you are an actual Jedi.