🔗 What is OpenResty XRay?

🔗 What is OpenResty XRay?

OpenResty XRay is a dynamic tracing product that can analyze various cloud and server applications in real-time. It treats running processes and containers as read-only databases and extracts necessary information to resolve performance issues, anomalies, errors, and security vulnerabilities. OpenResty XRay has a knowledge base, an inference engine, and hundreds of advanced analyzers that can diagnose and narrow down the root causes of deep issues without changing or affecting the target application.

🔗 Who should use OpenResty XRay?

DevOps can use OpenResty XRay to quickly discover any issues or identify the root causes, then forward the discovered issues and generate automatic reports to developers.

Development teams can also use OpenResty XRay to check for any regression issues during daily development or before the release of new software products.

🔗 Can OpenResty XRay analyze applications not running on OpenResty?

Of course! OpenResty XRay is a universal platform that can analyze any software. Well, at least the software you control (such as having access to DWARF debug symbols, source code, etc.).

However, OpenResty XRay currently works best with the OpenResty and Nginx technology stacks. Let us know which open-source software you would like OpenResty XRay to support next!

🔗 Does OpenResty XRay require any changes to my application?

No! OpenResty XRay uses advanced dynamic tracing technology and does not require cooperation from the target application. It is 100% non-intrusive.

1. We do not require plugins or modules to be added to your application.
2. We do not need to patch your application's code.
3. We do not require special compilation or startup options (when compiling C/C++ programs, you do need the -g option to send DWARF data, but this is very standard practice).
4. We do not inject any code into your application processes.

Using OpenResty XRay analyzers will not harm your application processes or even cause them to crash, as we do not add anything to your processes.

🔗 Is there a personal version available for individual use?

Yes, there is! As long as the personal version is not used for any company, government, or other commercial purposes. However, we usually do not provide SLA for the personal version.

🔗 Installation and Deployment

🔗 How do I deploy and install OpenResty XRay for my application?

You need to install and run the OpenResty XRay agent program on the server with your application or container. The agent program is a standalone daemon that can inspect all applications on the same server. It also acts as a client for the OpenResty XRay web console.

There are several ways to install the OpenResty XRay agent.

1. Install the RPM or Deb package repository we provide for the respective Linux distribution.
2. Install a large .tar.gz package containing all the RPM or Deb packages we provide for your Linux distribution.
3. Install our official Docker or Kubernetes (K8s) container images.

A common mistake is installing the agent in your own container. This will always fail because your container will almost always lack the permissions or filesystem access needed for the agent to inspect other processes and containers.

🔗 Which operating systems do we support?

Ubuntu 24.04 noble, Ubuntu 22.04 Jammy, Ubuntu 20.04 Focal, Ubuntu 18.04 Bionic, Debian 12 Bookworm, Debian 11 Bullseye, Debian 10 Buster, Debian 9 Stretch, Red Hat Enterprise Linux 8, Red Hat Enterprise Linux 7, Rocky Linux 9, Rocky Linux 8, AlmaLinux 9, AlmaLinux 8, Cent OS 8, Cent OS 7, Cent OS 6, Fedora 36, Fedora 35, Fedora 34, Fedora 33, Fedora 32, Amazon Linux 2023, Amazon Linux 2, Amazon Linux 1, Alibaba Cloud Linux 3, Alibaba Cloud Linux 2, Tencent Linux 3, Tencent Linux 2, OpenSUSE 15, Kylin 10, Kylin 7, CBL-Mariner 2

Our official Docker or K8s container images can run on any modern Linux distribution that supports containers, as long as the Linux distribution used within the target application containers is on the supported list.

If you want to run the OpenResty XRay agent service natively, we support most mainstream Linux distributions. Please check the latest distribution support list for details.

If you have a new Linux distribution that needs native support, please contact us: info@openresty.com!

In the future, we may also support non-Linux operating systems such as Android, macOS, *BSD, and Microsoft Windows.

🔗 Do I need a very new Linux kernel?

No! We support older kernels, such as the 3.10 kernel of CentOS 7. But we do not support older kernels like the 2.6 kernel of CentOS 6.

🔗 Can I use my own Linux kernel?

Yes! But you need to contact our team so that we can index your kernel in the software package database and thoroughly test its compatibility with our product. We do not yet support automatic indexing of custom or unknown kernels. We may consider adding this support in the future.

🔗 Is self-compiled kernel supported?

Yes, you need to upload the relevant kernel packages. The specific kernel packages required can be referenced on the upload page of the OpenResty XRay Console.

🔗 What applications and programming languages does OpenResty XRay support?

Supported programming languages and runtimes: Python, Java, Rust, Perl, PHP, Erlang, Ruby, Go, LuaJIT, C/C++ Supported servers and proxies: OpenResty, Envoy, Kong, CockroachDB, Nginx,

🔗 Can the OpenResty XRay agent analyze applications running on AWS EKS?

Yes.

🔗 Can I use OpenResty XRay on local virtual machines like VMWare/VirtualBox/KVM/etc.?

Yes! All such full-system virtualization technologies should work out of the box with OpenResty XRay. If not, please report it to us.

Please note that installing our agent inside your own Docker or K8s containers is not supported. But you can always use our official Docker and K8s container images to install the agent.

🔗 Can I run OpenResty XRay on public cloud virtual machines?

Yes! For example, Microsoft Azure, Google Cloud (GCP), Amazon Cloud (AWS), or Digital Ocean. Virtualized cloud servers are fine. Physical machines are not required.

🔗 Does the agent need a C compiler toolchain installed?

No. Unlike those open-source dynamic tracing frameworks like SystemTap or eBPF/BCC, OpenResty XRay does not require any C or C++ compiler toolchain to be installed on the target machine. No gcc, no clang, nothing. Also, it does not require any Linux kernel headers to be installed.

🔗 How is the OpenResty XRay agent deployed on K8s?

The OpenResty XRay Agent is usually deployed in a DaemonSet manner in Kubernetes environments.

🔗 Agent Usage and Management

🔗 How much overhead does the agent have?

We strive to minimize the performance impact of the agent program on your servers marked as "production". The agent uses an event-driven sampling method, so it does not constantly try to collect data, only when there is a real need. When not sampling, the additional overhead generated by all your processes is strictly 0. This is due to the truly non-intrusive nature of dynamic tracing. Even when sampling, the overhead of most of our analyzers should be less than 5% of your application. Even for those tools that are originally more expensive, the system will try to automatically adjust the frequency and process range of these analyzers to minimize the impact.

If you do not mark your servers as "production" in the web console, the system will consider them as development or test environments. Therefore, it will no longer be cautious about resource usage, and all analyzers will run at full speed so that you can get results as soon as possible.

🔗 Do I need to restart my processes after installing the agent?

No! One of the advantages of dynamic tracing technology is post-analysis. You do not need to restart your application or any existing processes for the newly installed OpenResty XRay agent to analyze. We can do this because we do not require any new plugins, new startup options, or code injection into your application. It is pollution-free and non-intrusive.

We understand that you may have some processes that have entered some rare states, and restarting them would make the problem disappear. This is why post-analysis and debugging are so critical.

🔗 How can I stop the agent?

To stop the OpenResty XRay agent service, simply run the following shell command.

sudo systemctl stop orxray-agent

But this is only temporary. If the system restarts or reboots, the agent service will start again. To stop it permanently, you can also run this command after running the previous command.

sudo systemctl disable orxray-agent

For containerized agent services, you can stop the agent container instance with Docker or Kubernetes.

🔗 How do I restart the agent?

To restart your agent, simply run the command.

sudo systemctl restart orxray-agent

Wait a few minutes, and the new agent instance will take effect.

🔗 My agent is not working, what should I do?

Sometimes, due to network or hard disk issues, the agent cannot work properly. In this case, check the error log files in the directory.

/usr/local/orxray-agent/logs/

If you need help with the errors in the error log files, please feel free to contact us.

🔗 How to run OpenResty XRay to analyze memory, CPU, etc.?

To run OpenResty XRay to analyze memory, CPU, etc., you can use its guided analysis features. For detailed operation steps and related cases, please refer to the series of articles on XRay online analysis of memory and high CPU issues on the official OpenResty blog.

🔗 Can the agent analyze private software?

Yes, if you have written private software. For example, if you do not have permission from other vendors, you should not analyze their proprietary software. However, if you have written the software or have access to the source code and have permission to compile it, then it is fine. You can even use our Y language (lang) to write advanced analyzers for your program. Make sure you compile private software with DWARF data (or "debug symbols"). For example, C/C++ programs should be compiled with the g or g3 option when using gcc or g++. Compiling with DWARF symbols has no impact on the runtime performance of the program, and most mainstream Linux distributions do this by default in their standard packages.

However, DWARF data is not always required. Some basic analyzers in OpenResty XRay do not require "debug symbols" to work. We once helped a customer accurately locate a memory leak in a third-party proprietary library at the C/C++ function level. But make sure you comply with third-party licensing requirements.

🔗 Data Security

🔗 What data does the agent collect?

The OpenResty XRay agent program only collects basic data used to analyze performance issues, troubleshoot stability and behavioral issues, and scan for security vulnerabilities. We never collect any potentially sensitive data from your business-level code. We also implement safeguards to prevent accidentally collecting sensitive data from data sources so that it does not leak outside your application processes.

🔗 How do you handle the data collected by the agent?

We are very careful to protect our customers' data and privacy. This is our top priority. When transmitting all data to our web console, we always use industry-standard encryption methods (such as TLS). Our web console's data storage and management also comply with all SOC 2 guidelines. Only the raw data of the last 2 weeks is stored in our web console (higher-level report data is stored for a longer period). If some customers have shorter data lifecycle requirements, we can also accommodate them.

🔗 Will you sell or share our data?

No, never. We will not sell your data to any third party. We may only share some audited and obfuscated data with some of our external contractors. And all our contractors must sign a non-disclosure agreement (NDA) with us.

🔗 Can I avoid uploading any data to your console?

Yes! We offer a self-hosted version of OpenResty XRay for companies with strict data handling policies. It is more expensive than the cloud version. For the On-Premise version, users deploy the web console in their own environment; therefore, no data is uploaded to OpenResty Inc.'s servers.

🔗 Debug Symbols and Analysis Capabilities

🔗 Do I need to install debug symbol packages on all my servers?

No. OpenResty XRay indexes all the debug symbols it finds in a remote database called package database. Whenever the agent finds a binary executable, it automatically queries the corresponding debug symbols in the package database.

We run crawlers to continuously obtain debug symbols (or DWARF data) for public software packages on the internet. We also automatically index users' own DWARF data and debug symbols. Each OpenResty XRay tenant has its own independent software package database to protect privacy. If you compile the software yourself, make sure you have an offline (or online) machine with all the software binaries and debug symbols installed and let the OpenResty XRay agent run there (at least for a while). Once the DWARF data and binary programs are indexed into the user's software package database, the user can get rid of the debug symbols on all their machines.

🔗 OpenResty XRay says my program lacks debug symbols, what should I do?

If the program is self-compiled, try to find the debug symbol package related to your main software package (such as *-debuginfo packages on RPM-based distributions, *-dbgsym.ddeb or *-dbgsym.deb files on Deb-based distributions). Install the debuginfo package on a machine with the agent installed and running, and wait for the agent to index the debug symbols.

Suppose your program was compiled without debug symbols, or you cannot find the corresponding debuginfo package. In that case, you can try recompiling your program with debug symbols enabled (for C/C++ programs, pass the -g or -g3 option in the compiler command line). Also, ensure your packaging scripts generate a debuginfo package (such as *-debuginfo*.rpm or *-dbgsym.deb) instead of discarding the debug symbols.

We understand that finding debug symbols can sometimes be a hassle. We are developing an AI system that can automatically reconstruct DWARF data for similar but not identical binary programs from existing DWARF data. Once this is ready, OpenResty XRay will be able to thoroughly analyze a large number of open-source software binaries that lack debug symbols.

It is worth mentioning that some basic analyzers in OpenResty XRay do not require "debug symbols" to work. In fact, we once helped a customer accurately locate a memory leak in a third-party proprietary library at the C/C++ function level.

🔗 Can I write my own analyzers?

Yes! You can use our Y language (or Ylang) to write your own analyzers. Ylang is a large subset of the C programming language with our own dynamic tracing extensions. We also provide the Ylua language, which is a subset of the Lua language with extensions, and YSQL, a SQL dialect.

🔗 Other Questions

🔗 Do these kernel messages have any impact?

🔗 vsyscall read attempt denied

After starting live recording, the kernel may throw messages like the one below. This message is harmless and can be ignored.

[29389.731303] redis-server[298169] vsyscall read attempt denied -- look up the vsyscall kernel parameter if you need a workaround ip:7f000013d241 cs:33 sp:2aaacbcf6f08 ax:0 si:7f00003db659 di:ffffffffff600000

🔗 How many accounts does 1 License support?

1 License supports 2 account logins.

🔗 How to configure HTTP proxy for OpenResty XRay?

If your OpenResty XRay Agent is deployed in an environment that requires accessing the public network through an HTTP proxy, you can configure and use the HTTP proxy by following the steps in this document.