Use Python Logging like a Professional
I can guess that just about each Python developer generally makes use of “print” for debugging. There’s nothing unsuitable with that for prototyping, however for manufacturing, there are far more efficient methods to deal with the logs. On this article, I’ll present 5 sensible explanation why Python “logging” is far more versatile and highly effective and why you completely ought to use it you probably have not began earlier than.
Let’s get into it.
Code
To make issues extra sensible, let’s think about a toy instance. I created a small utility that calculates a linear regression for 2 Python lists:
import numpy as np
from sklearn.linear_model import LinearRegression
from typing import Listing, Non-obligatory
def do_regression(arr_x: Listing, arr_y: Listing) -> Non-obligatory[List]:
“”” LinearRegression for X and Y lists “””
strive:
x_in = np.array(arr_x).reshape(-1, 1)
y_in = np.array(arr_y).reshape(-1, 1)
print(f”X: {x_in}”)
print(f”y: {y_in}”)
reg = LinearRegression().match(x_in, y_in)
out = reg.predict(x_in)
print(f”Out: {out}”)
print(f”Rating: {reg.rating(x_in, arr_y)}”)
print(f”Coef: {reg.coef_}”)
return out.reshape(-1).tolist()
besides ValueError as err:
print(f”ValueError: {err}”)
return None
if __name__ == “__main__”:
print(“App began”)
ret = do_regression([1,2,3,4], [5,6,7,8])
print(f”LinearRegression outcome: {ret}”)
This code works, however can we do it higher? We clearly can. Let’s see 5 benefits of utilizing “logging” as a substitute of “print” on this code.
1. Logging ranges
Let’s change our code a bit:
import logging
def do_regression(arr_x: Listing, arr_y: Listing) -> Non-obligatory[List]:
“””LinearRegression for X and Y Python lists”””
strive:
x_in = np.array(arr_x).reshape(-1, 1)
y_in = np.array(arr_y).reshape(-1, 1)
logging.debug(f”X: {x_in}”)
…
besides ValueError as err:
logging.error(f”ValueError: {err}”)
return None
if __name__ == “__main__”:
logging.basicConfig(stage=logging.DEBUG, format=’%(message)s’)
logging.data(“App began”)
ret = do_regression([1,2,3,4], [5,6,7,8])
logging.data(f”LinearRegression outcome: {ret}”)
Right here I changed “print” calls with “logging” calls. We made a small change, nevertheless it makes the output far more versatile. Utilizing the “stage” parameter, we are able to now set totally different logging ranges. For instance, if we use “stage=logging.DEBUG”, then all output will likely be seen. After we are certain that our code is prepared for manufacturing, we are able to change the extent to “logging.INFO”, and debugging messages won’t be displayed anymore:
And what’s necessary is that no code change is required besides the initialization of the logging itself!
By the way in which, all accessible constants will be discovered within the logging/__init__.py file:
ERROR = 40
WARNING = 30
INFO = 20
DEBUG = 10
NOTSET = 0
As we are able to see, the “ERROR” stage is the very best; by enabling the “ERROR” log stage, we are able to suppress all different messages, and solely errors will likely be displayed.
2. Formatting
As we are able to see from the final screenshot, it’s simple to regulate the logging output. However we are able to do far more to enhance that. We will additionally regulate the output by offering the “format” string. For instance, I can specify formatting like this:
logging.basicConfig(stage=logging.DEBUG,
format='[%(asctime)s] %(filename)s:%(lineno)d: %(message)s')
With out every other code modifications, I will see timestamps, file names, and even the road numbers within the output:
There are about 20 totally different parameters accessible, which will be discovered within the “LogRecord attributes” paragraph of the guide.
3. Saving logs to a file
Python logging is a really versatile module, and its performance will be simply expanded. Let’s say we need to save all our logs right into a file for future evaluation. To do that, we have to add solely two strains of code:
logging.basicConfig(stage=logging.DEBUG,
format='[%(asctime)s] %(message)s',
handlers=[logging.FileHandler("debug.log"),
logging.StreamHandler()])
As we are able to see, I added a brand new parameter “handlers”. A StreamHandler is displaying the log on the console, and the FileHandler, as we are able to guess from its identify, saves the identical output to the file.
This method is admittedly versatile. Loads of totally different “handler” objects can be found in Python, and I encourage readers to check the manual on their very own. And as we already know, logging works nearly mechanically; no additional code modifications are required.
4. Rotating log information
Saving logs right into a file is an effective choice, however alas, the disk house will not be limitless. We will simply remedy this downside by utilizing the rotating log file:
from logging.handlers import TimedRotatingFileHandler
…
if __name__ == “__main__”:
file_handler = TimedRotatingFileHandler(
filename=”debug.log”,
when=”midnight”,
interval=1,
backupCount=3,
)
logging.basicConfig(stage=logging.DEBUG,
format='[%(asctime)s] %(message)s’,
handlers=[file_handler, logging.StreamHandler()])
All parameters are self-explanatory. A TimedRotatingFileHandler object will create a log file, which will likely be modified each midnight, and solely the final three log information will likely be saved. The earlier information will likely be mechanically renamed to one thing like “debug.log.2023.03.03”, and after a 3-day interval, they are going to be deleted.
5. Sending logs by way of socket
Python’s logging is surprisingly versatile. If we don’t need to save logs into a neighborhood file, we are able to simply add a socket handler, which is able to ship logs to a different service utilizing a particular IP and port:
from logging.handlers import SocketHandler
logging.basicConfig(stage=logging.DEBUG, format='[%(asctime)s] %(message)s’,
handlers=[SocketHandler(host=”127.0.0.1″, port=15001),
logging.StreamHandler()])
That’s it; no extra code modifications are required!
We will additionally create one other utility that may take heed to the identical port:
import socket
import logging
import pickle
import struct
from logging import LogRecord
port = 15001
stream_handler = logging.StreamHandler()
def create_socket() -> socket.socket:
“””Create the socket”””
sock = socket.socket(socket.AF_INET)
sock.settimeout(30.0)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
return sock
def read_socket_data(conn_in: socket.socket):
“””Learn knowledge from socket”””
whereas True:
knowledge = conn_in.recv(4) # Knowledge: 4 bytes size + physique
if len(knowledge) > 0:
body_len = struct.unpack(“>L”, knowledge)[0]
knowledge = conn_in.recv(body_len)
report: LogRecord = logging.makeLogRecord(pickle.masses(knowledge))
stream_handler.emit(report)
else:
logging.debug(“Socket connection misplaced”)
return
if __name__ == “__main__”:
logging.basicConfig(stage=logging.DEBUG, format='[%(asctime)s] %(message)s’,
handlers=[stream_handler])
sock = create_socket()
sock.bind((“127.0.0.1”, port)) # Native connections solely
sock.hear(1) # One shopper will be related
logging.debug(“Logs listening thread began”)
whereas True:
strive:
conn, _ = sock.settle for()
logging.debug(“Socket connection established”)
read_socket_data(conn)
besides socket.timeout:
logging.debug(“Socket listening: no knowledge”)
The difficult half right here is to make use of the emit methodology, which provides all distant knowledge obtained by a socket to an energetic StreamHandler.
6. Bonus: Log filters
Lastly, a small bonus for readers who have been attentive sufficient to learn till this half. It’s also simple so as to add customized filters to logs. Let’s say we need to log solely X and Y values into the file for future evaluation. It’s simple to create a brand new Filter class, which is able to save to log solely strings containing “x:” or “y:” data:
from logging import LogRecord, Filter
class DataFilter(Filter):
“””Filter for logging messages”””
def filter(self, report: LogRecord) -> bool:
“””Save solely filtered knowledge”””
return “x:” in report.msg.decrease() or “y:” in report.msg.decrease()
Then we are able to simply add this filter to the file log. Our console output will keep intact, however the file may have solely “x:” and “y:” values.
file_handler = logging.FileHandler("debug.log")
file_handler.addFilter(DataFilter())
logging.basicConfig(stage=logging.DEBUG,
format='[%(asctime)s] %(message)s’,
handlers=[file_handler, logging.StreamHandler()])
Conclusion
On this quick article, we discovered a number of simple methods to include logs into the Python utility. Logging in Python is a really versatile framework, and it’s undoubtedly price spending a while investigating the way it works.
Thanks for studying, and good luck with future experiments.
Should you loved this story, be at liberty to subscribe to Medium, and you’re going to get notifications when my new articles will likely be printed, in addition to full entry to 1000’s of tales from different authors.