Compiled code contains too many tokens

This compilation error indicates that a script’s compiled code is too large for the runtime system to execute.

Before a script executes on a dataset, it is compiled by the Pine Script compiler. During this process, the compiler translates the written source code into a tokenized intermediate language (IL), then generates the final compiled code from the script’s IL form. Translating a script into an IL representation enables the compiler to perform various internal transformations to optimize the script’s structure.

Pine Script limits the maximum size of each indicator, strategy, and library for resource efficiency. This limit applies to the total size of the script’s IL translation after optimization. It does not apply to the number of characters or code lines in the original written code. This behavior explains why one script might fail with the “too many tokens” error during compilation, while another with the same number of characters or lines compiles and runs successfully. It also explains why some kinds of manual code changes, such as removing comments or shortening variable names, do not affect the size of a compiled script.

The maximum possible size of a script’s IL translation is 100,256 tokens. There are several ways to reduce a translated script’s size if it exceeds this limit, depending on the original script’s structure and the kinds of code it contains.

The sections on this page explain some of the typical optimizations that the compiler performs before measuring a translated script’s size, various types of manual code changes that have little to no effect on a script’s translation, and the most common techniques that can help reduce script size and resolve this error.

Ineffective code changes

Before the compiler measures the size of a translated script, it performs several code transformations to eliminate unnecessary tokens and optimize the script’s structure. Because of these transformations, several types of source code changes a programmer might consider to resolve the “too many tokens” error have little to no impact on the size of the compiled code. Common types of ineffective code changes include the following:

• Revising or removing comments
• Removing whitespaces and merging code lines
• Changing code identifiers
• Swapping constant expressions and values
• Reducing identical expressions and functions
• Removing unused code

The following sections explain why these common source code changes do not help resolve this error. To learn about code changes that can meaningfully reduce the size of a translated script, refer to the Effective techniques to reduce script size section below.

Revising or removing comments

Programmers might assume that reducing the length of the comments in their scripts or removing them altogether reduces the size of the compiled code. However, the compiler discards all comments during translation, because their main purpose is to provide extra information in the source code for script readers. Therefore, reducing or removing comments has no effect on the compiled code.

For example, the simple script below contains multiple comments that document what each part of the code does:

If we remove all the comments from the script, the source code becomes visibly shorter. However, because the compiler excludes comments automatically, the version of the code below translates to the same compiled result as the previous code:

Removing whitespaces and merging code lines

Pine Script code uses whitespace characters, including standard spaces and line terminators, to separate expressions and statements, organize code on multiple text lines, and visually delimit code blocks. However, the IL translation of a script uses a compact format, without most whitespace delimiters, to represent the script’s structure. Therefore, removing spaces between items in the source code does not affect the script’s compiled size. Likewise, the size is not affected by merging multiple statements onto a single code line and separating them with commas.

For example, the following script lists statements across multiple lines of code. The source code includes space separators, line-wrapped expressions, and multiple blank lines:

The modified script version below removes multiple space separators and blank lines, eliminates line wrapping, and groups the statements onto fewer lines. Although the code is visually shorter and more compact, its IL form is identical to that of the previous version, because the compiler already discards the spaces and line terminators automatically while translating the code:

Note that:

• Automatic whitespace removal does not apply to literal strings, because each character in a string is part of the saved value. The values of literal strings and other “string” constants are embedded directly into a script’s IL translation and do affect the total size. See the Shortening or removing constant strings section below to learn more.

Changing code identifiers

When the compiler translates a script to an IL form, it automatically replaces the identifiers (names) of declared variables, user-defined functions, function parameters, user-defined types, and type fields with compact internal identifiers based on the code’s structure. These identifiers do not vary with the variable, function, and type names specified by the programmer in the source code. Therefore, changing identifiers in a script does not affect the script’s compiled size.

Similarly, named arguments in function calls (e.g., f(x = 1)) are automatically converted to positional arguments (e.g., f(1)) when possible. Therefore, manually converting named arguments to positional arguments typically has little to no effect on the compiled code’s size.

The following example script contains a user-defined function with multiple verbose identifiers in its signature and body. The script calls this function and the plot() function using named arguments. It also declares multiple variables with lengthy identifiers:

The script version below reduces the verbosity of all the script’s identifiers and changes the function calls to use positional arguments instead of named arguments. Although the code is now visibly more compact, it still translates to the same IL form as the previous version:

Note that:

• We also removed type keywords from most of the variable and parameter declarations. If a declaration does not include a type keyword, the compiler automatically infers the appropriate type based on how the script uses the variable or parameter. Therefore, omitting type keywords typically does not affect the size of the IL translation.

Swapping constant expressions and values

When possible, the compiler automatically evaluates expressions that return constant values during translation. Rather than including tokens to represent each part of a constant expression in the script’s IL form, the compiler embeds the expression’s final value into the translated code. This optimization prevents the script from repeatedly recalculating the result at runtime. Therefore, replacing a constant expression with a literal value in the source code typically does not affect the size of the compiled script.

Similarly, if an if or switch structure or a conditional expression relies on a constant true or false condition, the compiler often simplifies the logic by replacing it with tokens for only the calculations that the script consistently executes. Therefore, manually simplifying such logic has little to no effect on the compiled script’s size.

The following example script assigns an expression that returns a “const int” value to a variable. It then checks the expression fixedValue == 6 in an if structure and adds 10 to the variable’s value if the expression evaluates to true:

The above script plots a constant value of 16. The fixedValue variable is initially assigned a value of 6, so the fixedValue == 6 condition always evaluates to true, thus causing the variable’s final value to change to 16 on each bar. The compiler recognizes this pattern in the code and simplifies the logic to fixedValue = 16. Therefore, the script has the same IL translation as the script below:

Reducing identical expressions and functions

If the compiler detects that multiple expressions are structurally identical and consistently evaluate to the same value, it automatically reduces the duplicate expressions to avoid repetitive calculations when possible. Rather than including tokens for each repetition, the compiler assigns the translated expression to an internal variable, then uses that variable in all code that requires the expression’s result. This optimization helps conserve compiled tokens and potentially reduce the script’s runtime.

Therefore, manually reducing the repetition of expressions that the compiler can automatically replace has little to no effect on the size of a compiled script.

The following script shows a scenario in which removing repeated expressions does not affect the size of the compiled code. The script defines a posPercent() function, which calculates the percentage of positive values in a series over a specified length. The script plots the results of seven calls to the function, each using identical arguments:

As the compiler translates the script, it verifies that all seven posPercent() expressions use identical value-type arguments, perform operations in the same order, and return the same result. It then restructures the script to execute only one call to the function and reuse the result in all the plot() calls. Therefore, the script has the same IL translation as the following script, which assigns one posPercent() call to a single variable and plots the variable’s value seven times:

Similar to the above, if the compiler identifies user-defined functions or methods that are structurally identical, have the same parameter types, and consistently return matching results when passed the same arguments, it automatically removes the duplicate function definitions from the translated code rather than including tokens for each one, when possible. The compiled script then treats each call to a duplicate function as an alias for a call to the first function. Therefore, manually removing such duplicate functions from the code does not typically affect the compiled script’s size.

To illustrate this behavior, the following script includes the posPercent() function from our previous example, along with a separate function named f() that follows the same structure and differs only in its identifiers. The script executes one posPercent() call and one f() call, then plots the results:

Aside from differences in identifiers, both functions in the script are identical. They have the same required parameters, rely on the same value types, perform calculations in the same order, and return matching values when passed the same arguments. The compiler recognizes this pattern and includes tokens for only the first function definition in the IL translation. After this transformation, the f() call is treated as a call to posPercent(). Therefore, the above script has the same compiled form as the script below:

Note that:

• If we profile the previous script that contains an f() call, the Pine Profiler does not display any performance data next to the lines in that function’s definition. Instead, it shows combined performance results for both function calls next to the posPercent() definition, confirming that the duplicate function is removed during translation.

Removing unused code

As the compiler translates a script, it verifies which parts of the code affect the script’s outputs, such as plots, drawings, strategy orders, alerts, and Pine Logs. If the script contains code that does not affect any outputs, the compiler recognizes that the code is unused and automatically discards tokens for it in the IL translation. Therefore, manually removing unused code does not reduce a script’s compiled size.

The following example script contains a user-defined function that creates an array of simple moving averages over a range of specified lengths. It calls the function to create an array containing averages of close values over 2 to 50 bars, then uses the results to calculate a custom oscillator. However, the script does not use this code in any outputs; the only output from this script is a single plot(close) call:

The compiler recognizes that the user-defined function and all related calculations are unused by the script’s outputs, so it excludes tokens for that code in the IL translation. Consequently, after this transformation, the script above has the same compiled result as the following script:

Effective techniques to reduce script size

The following sections list some of the most common types of code changes that can help meaningfully reduce the size of a compiled script. These techniques are effective because they involve code revisions that the compiler cannot perform automatically during translation. The techniques covered below include the following:

• Shrinking or removing embedded data
• Shortening or removing constant strings
• Replacing unrolled statements with loops
• Encapsulating code in functions
• Replacing custom code with built-ins
• Splitting the script into parts

Shrinking or removing embedded data

One of the most common causes of the “too many tokens” error is a large quantity of embedded data in the translated code. A typical cause of large embedded data is the use of numerous constant values in calls to functions that support arbitrary numbers of arguments, such as array.from().

The Pine Script compiler embeds every literal “int”, “float”, “bool”, and “string” value used in a source code into the script’s IL translation. Likewise, it automatically evaluates constant expressions of these types, then embeds their results in the translated code to prevent recalculation at runtime. Additionally, if a script uses constant variables or expressions in function or method calls, the compiler inlines their values directly into each call site.

Therefore, hard-coding numerous constants into a script can significantly increase its compiled size, especially if the script uses them as arguments in function calls. For instance, a single array.from() call with thousands of constant arguments can consume the entire size budget on its own, depending on the saved values.

To reduce the amount of embedded data in a compiled script:

• Swap large constants with concise expressions that compute the same values at runtime where possible.
• Replace function calls and other expressions that use long lists of constants with logic that yields equivalent results at runtime. For instance, rather than defining an array.from() call with numerous constant arguments, create a loop that iteratively calculates the elements and inserts them into the array using functions such as array.set() or array.push().
• If using a lengthy function call or other expression with many hard-coded values is unavoidable, consider moving the call into a user-defined function and exporting that function from a separate library. See the Splitting the script into parts section to learn more.

The following example script calculates a weighted moving average with custom precomputed weights. The script uses a single array.from() call with multiple literal “float” arguments to create an array of weights. It populates a single-row matrix with the array’s elements, calls the matrix.mult() method to calculate a weighted sum, then divides the result by the sum of the weights to compute the moving average. The array.from() call in this script significantly increases the size of the compiled code, because each literal value used as an argument is embedded directly into the IL translation. Adding multiple similar calls to this script can easily cause the “too many tokens” error:

The array in this example contains a series of square numbers, where the first element is 1 (1 * 1), the second is 4 (2 * 2), and so on. In other words, each element is the square of i + 1, where i refers to the element’s index. Therefore, we can reduce our script’s size by calculating the array’s values within a loop rather than defining them directly in the code.

The script version below creates a persistent array using an array.new<float>() call, then uses a loop to populate it on the first bar. On each iteration, the script calculates a square value, then calls the array.push() function to push the value into the array. This version calculates the same results as the previous script, but its compiled size is substantially smaller because we avoided defining a long list of literal values. Additionally, this source code is visibly shorter and simpler to maintain:

Shortening or removing constant strings

As explained in the section Shrinking or removing embedded data above, the compiler embeds constant values, including strings, into a script’s IL translation where possible. Consequently, using lengthy constant strings in a script can significantly increase the size of the compiled code, especially if the script passes the strings as multiple arguments in function calls.

If a script with lengthy or multiple constant strings causes the “too many tokens” error, try any of the following to reduce the script’s compiled size:

• Remove all lengthy constant strings that the script’s logic does not absolutely require, and try to reduce the length of all others.
• Avoid using a long constant string for multiple arguments in function calls. Each instance of the argument’s value is inlined at the call site when possible. Therefore, using the value for multiple arguments can multiply its effect on the compiled code’s size.
• Check for repetitive expressions that use the same constant string. If possible, replace the repeated expressions with a single expression that executes in a loop. For instance, if the script contains multiple str.format() calls with the same formatting string, replace them with a loop that executes a single str.format() call multiple times.
• Shorten the titles of enum members used by the script’s logic. The titles of enum members are embedded into parts of the code that require their values. By default, the title of each member is the “const string” representation of its name. Therefore, if an enum member does not have a specified title, shorten the member’s name or assign it a short title.
• If the script requires a specific lengthy string, but does not require it to be a constant, create a user-defined function that returns a “simple” or “series” version of the value, or the ID of a collection that contains the value, then export the function from a separate library. See the Splitting the script into parts section to learn more.

Replacing unrolled statements with loops

Loop unrolling is a common technique in which the programmer writes a set of repetitive statements and expressions to perform a sequence of calculations without a loop. This technique can marginally improve a script’s runtime in some cases. However, each instance of repeated code that the compiler cannot reduce adds tokens to the script’s IL translation. Consequently, writing several pieces of unrolled code can impact the size of a compiled script.

Therefore, if a script contains multiple unrolled statements or expressions, defining an equivalent, concise loop to execute the necessary calculations can help reduce the script’s compiled size.

For example, the following script calculates the number of close values that are greater than the current value over the past 50 bars. The script calculates the value using 50 written statements that perform the same operations using different values. All of the statements are unique and cannot be reduced by the compiler. Therefore, each one adds tokens to the script’s IL translation:

We can reduce our script’s token count by replacing all the repetitive statements with a single loop that executes the required logic iteratively. The script version below uses a for loop that increments its counter variable (i) from 1 to 50. The single statement in the loop’s local block increments the higherCloses value on each iteration where the value of close[i] is greater than the close value on the current bar. This version of the script achieves the same result as the previous version, but its compiled size is significantly smaller:

Encapsulating code in functions

Function calls in Pine Script are not inlined, meaning the compiler does not replace each function call with a copy of the function’s body during translation, unlike the compilers for some other languages. However, the compiler does typically include tokens for each argument in a function call, including the function’s default arguments. Therefore, the size that each function call contributes to a compiled script often varies with the number of parameters.

Calls to functions with few parameters typically have low impact on the size of the compiled code, unless the arguments are lengthy constant strings. By contrast, calls to functions with many parameters, or to functions that support an arbitrary number of arguments, can significantly impact a compiled script’s size in some cases. See the Shrinking or removing embedded data section above for an example.

If a script contains multiple large, repetitive statements or expressions, or calls to functions with many parameters, encapsulating that code in a separate user-defined function containing as few parameters as possible can help reduce the size of a compiled script.

The following example script calculates an average Efficiency Ratio across several periods that are powers of two. The script declares a variable with an initial value of 0, then uses nine addition assignment operations that use the same expressions but different length values. It then divides the value by 9 and plots the result. Each of the repetitive statements in this script contributes multiple tokens to the script’s IL translation. We cannot simplify the script by replacing the statements with a loop, because the ta.change() and math.sum() calls require one execution on every bar for correct results:

We can reduce our script’s size by encapsulating the structure of the repetitive expressions within a user-defined function, then replacing each expression with a concise function call. The script version below defines a calcER() function, which calculates the Efficiency Ratio of close values over a specified length. It then uses a call to this function instead of the repeated expressions in each addition assignment operation. Each calcER() call is not inlined; the compiler adds tokens only for the function’s reference and its single argument. Therefore, in addition to being visibly more compact, this version has a smaller compiled size:

Replacing custom code with built-ins

Pine Script features many built-in functions and variables that help simplify code and streamline script creation. Each written call to a built-in function produces compiled tokens for only the function’s reference and the call’s arguments. Similarly, many built-in variables produce minimal tokens for referencing data calculated outside the script.

Sometimes, programmers create custom code that mirrors built-ins or yields equivalent results. Custom code implementations provide extra freedom to modify calculations as needed. However, they also typically contribute more tokens to a script’s IL translation. Therefore, if a script performs calculations that can be done with built-ins, replacing those calculations with built-ins can often help reduce a script’s compiled size.

The following example script computes a weighted moving average of hlc3 values over an input length, and it calculates the daily volume-weighted average price. The script uses custom functions to compute both values rather than using equivalent built-ins. All the code defined within those functions adds tokens to the script’s IL translation, thus increasing its size:

Implementing custom functions for our script’s calculations is unnecessary. We can use the built-in ta.wma() function to calculate the weighted moving average, and use the ta.vwap variable to retrieve the daily volume-weighted average price. The script version below implements these changes. This version computes the same results, is visibly shorter, and has a smaller compiled size:

Splitting the script into parts

If a script causes the “too many tokens” error, and no manual source code revisions resolve it without compromising the script’s logic, the typical solution is to split the code into two or more smaller scripts or offload reusable code components into libraries. Distributing a script’s logic across multiple scripts directly reduces the compiled tokens from a single source code.

Programmers can try any of the following to split a large script’s logic, depending on the types of code it uses and the tasks that it performs:

• If the script contains multiple large function or method definitions, consider exporting them from a separate library, and importing that library into the script. Likewise, if the script contains several lengthy statements, consider defining exported library functions to encapsulate those statements, if possible. Moving functions to libraries often fixes the problem, because libraries are compiled independently and their code does not directly affect the compiled size of the scripts that import them. A single script can import multiple libraries, provided their combined size does not exceed 1 million tokens. Refer to the Libraries page to learn more about library scripts and how to use them.
• If the script performs multiple independent tasks, create independent scripts for each task instead. For example, if a script calculates multiple complex indicators for no reason other than to consolidate them into a single source code, move each indicator’s logic into a separate script to keep compiled sizes low.
• If the script performs a lengthy set of related calculations, consider creating separate indicators for intermediate stages of those calculations, then use plots and source inputs to pass data between them. To learn more about connecting scripts to other scripts, refer to the How to apply an indicator or strategy to another indicator article in our Help Center. Note that charts allow up to 24 total indicator-on-indicator connections, and a maximum of 11 indicators connected sequentially (i.e., indicator_1 -> indicator_2 -> ... indicator_11). See this related article in our Help Center for more information.