_
    _ __  _ __ ___         __ _ _ __ __ _ _ __ | |__
   | '_ \| '_ ` _ \ _____ / _` | '__/ _` | '_ \| '_ \
   | |_) | | | | | |_____| (_| | | | (_| | |_) | | | |
   | .__/|_| |_| |_|      \__, |_|  \__,_| .__/|_| |_|
   |_|                    |___/          |_|

   pm-graph: suspend/resume/boot timing analysis tools
    Version: 5.11
     Author: Todd Brandt <todd.e.brandt@intel.com>
  Home Page: https://www.intel.com/content/www/us/en/developer/topic-technology/open/pm-graph/overview.html

 Report bugs/issues at bugzilla.kernel.org Tools/pm-graph
	- https://bugzilla.kernel.org/buglist.cgi?component=pm-graph&product=Tools

 Full documentation available online & in man pages
	- Getting Started:
	  https://www.intel.com/content/www/us/en/developer/articles/technical/usage.html

	- Feature Summary:
	  https://www.intel.com/content/www/us/en/developer/topic-technology/open/pm-graph/features.html

	- upstream version in git:
	  git clone https://github.com/intel/pm-graph/

 Table of Contents
	- Overview
	- Setup
	- Usage
		- Basic Usage
		- Dev Mode Usage
		- Proc Mode Usage
	- Endurance Testing
		- Usage Examples
	- Configuration Files
		- Usage Examples
		- Config File Options
	- Custom Timeline Entries
		- Adding/Editing Timeline Functions
		- Adding/Editing Dev Timeline Source Functions
		- Verifying your Custom Functions
	- Testing on consumer linux Operating Systems
		- Android

------------------------------------------------------------------
|                          OVERVIEW                              |
------------------------------------------------------------------

 This tool suite is designed to assist kernel and OS developers in optimizing
 their linux stack's suspend/resume & boot time. Using a kernel image built
 with a few extra options enabled, the tools will execute a suspend or boot,
 and will capture dmesg and ftrace data. This data is transformed into a set of
 timelines and a callgraph to give a quick and detailed view of which devices
 and kernel processes are taking the most time in suspend/resume & boot.

------------------------------------------------------------------
|                            SETUP                               |
------------------------------------------------------------------

    Package Requirements
       - runs with python2 or python3, choice is made by /usr/bin/python link
       - python
       - python-configparser (for python2 sleepgraph)
       - python-requests (for stresstester.py)
       - linux-tools-common (for turbostat usage in sleepgraph)

       Ubuntu:
          sudo apt-get install python python-configparser python-requests linux-tools-common

       Fedora:
          sudo dnf install python python-configparser python-requests linux-tools-common

    The tools can most easily be installed via git clone and make install

    $> git clone http://github.com/intel/pm-graph.git
    $> cd pm-graph
    $> sudo make install
    $> man sleepgraph ; man bootgraph

    Setup involves some minor kernel configuration

    The following kernel build options are required for all kernels:
        CONFIG_DEVMEM=y
        CONFIG_PM_DEBUG=y
        CONFIG_PM_SLEEP_DEBUG=y
        CONFIG_FTRACE=y
        CONFIG_FUNCTION_TRACER=y
        CONFIG_FUNCTION_GRAPH_TRACER=y
        CONFIG_KPROBES=y
        CONFIG_KPROBES_ON_FTRACE=y

	In kernel 3.15.0, two patches were upstreamed which enable the
        v3.0 behavior. These patches allow the tool to read all the
        data from trace events instead of from dmesg. You can enable
        this behavior on earlier kernels with these patches:

        (kernel/pre-3.15/enable_trace_events_suspend_resume.patch)
        (kernel/pre-3.15/enable_trace_events_device_pm_callback.patch)

	If you're using bootgraph, or sleepgraph with a kernel older than 3.15.0,
		the following additional kernel parameters are required:
        (e.g. in file /etc/default/grub)
        GRUB_CMDLINE_LINUX_DEFAULT="... initcall_debug log_buf_len=32M ..."

	If you're using a kernel older than 3.11-rc2, the following simple
		patch must be applied to enable ftrace data:
        in file: kernel/power/suspend.c
        in function: int suspend_devices_and_enter(suspend_state_t state)
        remove call to "ftrace_stop();"
        remove call to "ftrace_start();"

        There is a patch which does this for kernel v3.8.0:
        (kernel/pre-3.11-rc2/enable_ftrace_in_suspendresume.patch)



------------------------------------------------------------------
|                            USAGE                               |
------------------------------------------------------------------

Basic Usage
___________

 1) First configure a kernel using the instructions from the previous sections.
    Then build, install, and boot with it.
 2) Open up a terminal window and execute the mode list command:

	%> sudo ./sleepgraph.py -modes
		['freeze', 'mem', 'disk']

 Execute a test using one of the available power modes, e.g. mem (S3):

	%> sudo ./sleepgraph.py -m mem -rtcwake 15

		or with a config file

	%> sudo ./sleepgraph.py -config config/suspend.cfg

 When the system comes back you'll see the script finishing up and
 creating the output files in the test subdir. It generates output
 files in subdirectory: suspend-mmddyy-HHMMSS. The ftrace file can
 be used to regenerate the html timeline with different options

     HTML output:                    <hostname>_<mode>.html
     raw dmesg output:               <hostname>_<mode>_dmesg.txt
     raw ftrace output:              <hostname>_<mode>_ftrace.txt

 View the html in firefox or chrome.


Dev Mode Usage
______________

 Developer mode adds information on low level source calls to the timeline.
 The tool sets kprobes on all delay and mutex calls to see which devices
 are waiting for something and when. It also sets a suite of kprobes on
 subsystem dependent calls to better fill out the timeline.

 The tool will also expose kernel threads that don't normally show up in the
 timeline. This is useful in discovering dependent threads to get a better
 idea of what each device is waiting for. For instance, the scsi_eh thread,
 a.k.a. scsi resume error handler, is what each SATA disk device waits for
 before it can continue resume.

 The timeline will be much larger if run with dev mode, so it can be useful
 to set the -mindev option to clip out any device blocks that are too small
 to see easily. The following command will give a nice dev mode run:

 %> sudo ./sleepgraph.py -m mem -rtcwake 15 -mindev 1 -dev

	or with a config file

 %> sudo ./sleepgraph.py -config config/suspend-dev.cfg


Proc Mode Usage
_______________

 Proc mode adds user process info to the timeline. This is done in a manner
 similar to the bootchart utility, which graphs init processes and their
 execution as the system boots. This tool option does the same thing but for
 the period before and after suspend/resume.

 In order to see any process info, there needs to be some delay before or
 after resume since processes are frozen in suspend_prepare and thawed in
 resume_complete. The predelay and postdelay args allow you to do this. It
 can also be useful to run in x2 mode with an x2 delay, this way you can
 see process activity before and after resume, and in between two
 successive suspend/resumes.

 The command can be run like this:

 %> sudo ./sleepgraph.py -m mem -rtcwake 15 -x2 -x2delay 1000 -predelay 1000 -postdelay 1000 -proc

	or with a config file

 %> sudo ./sleepgraph.py -config config/suspend-proc.cfg

------------------------------------------------------------------
|                     ENDURANCE TESTING                          |
------------------------------------------------------------------

 The best way to gauge the health of a system is to run a series of
 suspend/resumes over an extended period and analyze the behavior. This can be
 accomplished with sleepgraph's -multi argument. You specify two numbers: the
 number of tests to run OR the duration in days, hours, or minutes, and the
 delay in seconds between them. For instance, -multi 20 5: execute 20 tests with
 a 5 second delay between each, or -multi 24h 0: execute tests over a 24 hour
 period with no delay between tests. You can include any other options you like
 to generate the data you want. It's most useful to collect dev mode timelines
 as the kprobes don't alter the performance much and you get more insight.

 On completion, the output folder contains a series of folders for the
 individual test data and a set of summary pages in the root. The summary.html
 file is a tabular list of the tests with relevant info and links. The
 summary-issue.html and summary-devices.html files include data taken from
 all tests on kernel issues and device performance. The folder looks like this:

  suspend-xN-{date}-{time}:
	summary.html
	summary-issues.html
	summary-devices.html
	suspend-{date}-{time} (1)
	suspend-{date}-{time} (2)
	...

 These are the relevant arguments to use for testing:

  -m mode
	Mode to initiate for suspend e.g. mem, freeze, standby (default: mem).

  -rtcwake t
	Use rtcwake to autoresume after t seconds (default: 15).

  -gzip (optional)
	Gzip the trace and dmesg logs to save space. The tool can also read in
	gzipped logs for processing. This reduces the multitest folder size.

  -dev (optional)
	Add kernel source calls and threads to the timeline (default: disabled).

  -multi n d
	Execute n consecutive tests at d seconds intervals. The outputs will be
	created in a new subdirectory: suspend-xN-{date}-{time}. When the multitest
	run is done, the -summary command is called automatically to create summary
	html files for all the data (unless you use -skiphtml). -skiphtml will
	speed up the testing by not creating timelines or summary html files. You
	can then run the tool again at a later time with -summary and -genhtml to
	create the timelines.

  -skiphtml (optional)
	Run the test and capture the trace logs, but skip the timeline and summary
	html generation. This can greatly speed up overall testing. You can then
	copy the data to a faster host machine and run -summary -genhtml to
	generate the timelines and summary.

 These are the relevant commands to use after testing is complete:

  -summary indir
	Generate or regenerate the summary for a -multi test run. Creates three
	files: summary.html, summary-issues.html, and summary-devices.html in the
	current folder. summary.html is a table of tests with relevant info sorted
	by kernel/host/mode, and links to the test html files. summary-issues.html
	is a list of kernel issues found in dmesg from all the tests.
	summary-devices.html is a list of devices and times from all the tests.

  -genhtml
	Used  with -summary to regenerate any missing html timelines from their
	dmesg and ftrace logs. This will require a significant amount of time if
	there are thousands of tests.

Usage Examples
_______________

 A multitest is initiated like this:

  %> sudo ./sleepgraph.py -m mem -rtcwake 10 -dev -gzip -multi 2000 0

	or you can skip timeline generation in order to speed things up

  %> sudo ./sleepgraph.py -m mem -rtcwake 10 -dev -gzip -multi 2000 0 -skiphtml

 The tool will produce an output folder with all the test subfolders inside.
 Each test subfolder contains the dmesg/ftrace logs and/or the html timeline
 depending on whether you used the -skiphtml option. The root folder contains
 the summary.html files.

 The summary for an existing multitest is generated like this:

  %> cd suspend-x2000-{date}-{time}
  %> sleepgraph.py -summary .

	or if you need to generate the html timelines you can use -genhtml

  %> cd suspend-xN-{date}-{time}
  %> sleepgraph.py -summary . -genhtml

------------------------------------------------------------------
|                    CONFIGURATION FILES                         |
------------------------------------------------------------------

 Since 4.0 we've moved to using config files in lieu of command line options.
 The config folder contains a collection of typical use cases.
 There are corresponding configs for other power modes:

	Simple suspend/resume with basic timeline (mem/freeze/standby)
		config/suspend.cfg
		config/freeze.cfg
		config/standby.cfg

	Dev mode suspend/resume with dev timeline (mem/freeze/standby)
		config/suspend-dev.cfg
		config/freeze-dev.cfg
		config/standby-dev.cfg

	Simple suspend/resume with timeline and callgraph (mem/freeze/standby)
		config/suspend-callgraph.cfg
		config/freeze-callgraph.cfg
		config/standby-callgraph.cfg

	Sample proc mode x2 run using mem suspend
		config/suspend-x2-proc.cfg

	Sample for editing timeline funcs (moves internal functions into config)
		config/custom-timeline-functions.cfg

	Sample debug config for serio subsystem
		config/debug-serio-suspend.cfg


Usage Examples
______________

 Run a simple mem suspend:
 %> sudo ./sleepgraph.py -config config/suspend.cfg

 Run a mem suspend with callgraph data:
 %> sudo ./sleepgraph.py -config config/suspend-callgraph.cfg

 Run a mem suspend with dev mode detail:
 %> sudo ./sleepgraph.py -config config/suspend-dev.cfg


Config File Options
___________________

 [Settings]

 # Verbosity: print verbose messages (def: false)
 verbose: false

 # Suspend Mode: e.g. standby, mem, freeze, disk (def: mem)
 mode: mem

 # Output Directory Format: {hostname}, {date}, {time} give current values
 output-dir: suspend-{hostname}-{date}-{time}

 # Automatic Wakeup: use rtcwake to wakeup after X seconds (def: infinity)
 rtcwake: 15

 # Add Logs: add the dmesg and ftrace log to the html output (def: false)
 addlogs: false

 # Sus/Res Gap: insert a gap between sus & res in the timeline (def: false)
 srgap: false

 # Custom Command: Command to execute in lieu of suspend (def: "")
 command: echo mem > /sys/power/state

 # Proc mode: graph user processes and cpu usage in the timeline (def: false)
 proc: false

 # Dev mode: graph source functions in the timeline (def: false)
 dev: false

 # Suspend/Resume x2: run 2 suspend/resumes back to back (def: false)
 x2: false

 # x2 Suspend Delay: time delay between the two test runs in ms (def: 0 ms)
 x2delay: 0

 # Pre Suspend Delay: nclude an N ms delay before (1st) suspend (def: 0 ms)
 predelay: 0

 # Post Resume Delay: include an N ms delay after (last) resume (def: 0 ms)
 postdelay: 0

 # Min Device Length: graph only dev callbacks longer than min (def: 0.001 ms)
 mindev: 0.001

 # Callgraph: gather ftrace callgraph data on all timeline events (def: false)
 callgraph: false

 # Expand Callgraph: pre-expand the callgraph treeviews in html (def: false)
 expandcg: false

 # Min Callgraph Length: show callgraphs only if longer than min (def: 1 ms)
 mincg: 1

 # Timestamp Precision: number of sig digits in timestamps (0:S, [3:ms], 6:us)
 timeprec: 3

 # Device Filter: show only devs whose name/driver includes one of these strings
 devicefilter: _cpu_up,_cpu_down,i915,usb

 # Override default timeline entries:
 # Do not use the internal default functions for timeline entries (def: false)
 # Set this to true if you intend to only use the ones defined in the config
 override-timeline-functions: true

 # Override default dev timeline entries:
 # Do not use the internal default functions for dev timeline entries (def: false)
 # Set this to true if you intend to only use the ones defined in the config
 override-dev-timeline-functions: true

 # Call Loop Max Gap (dev mode only)
 # merge loops of the same call if each is less than maxgap apart (def: 100us)
 callloop-maxgap: 0.0001

 # Call Loop Max Length (dev mode only)
 # merge loops of the same call if each is less than maxlen in length (def: 5ms)
 callloop-maxlen: 0.005

------------------------------------------------------------------
|                   CUSTOM TIMELINE ENTRIES                      |
------------------------------------------------------------------

Adding or Editing Timeline Functions
____________________________________

 The tool uses an array of function names to fill out empty spaces in the
 timeline where device callbacks don't appear. For instance, in suspend_prepare
 the tool adds the sys_sync and freeze_processes calls as virtual device blocks
 in the timeline to show you where the time is going. These calls should fill
 the timeline with contiguous data so that most kernel execution is covered.

 It is possible to add new function calls to the timeline by adding them to
 the config. It's also possible to copy the internal timeline functions into
 the config so that you can override and edit them. Place them in the
 timeline_functions_ARCH section with the name of your architecture appended.
 i.e. for x86_64: [timeline_functions_x86_64]

 Use the override-timeline-functions option if you only want to use your
 custom calls, or leave it false to append them to the internal ones.

 This section includes a list of functions (set using kprobes) which use both
 symbol data and function arg data. The args are pulled directly from the
 stack using this architecture's registers and stack formatting. Each entry
 can include up to four pieces of info: The function name, a format string,
 an argument list, and a color. But only a function name is required.

 For a full example config, see config/custom-timeline-functions.cfg. It pulls
 all the internal timeline functions into the config and allows you to edit
 them.

  Entry format:

    function: format{fn_arg1}_{fn_arg2} fn_arg1 fn_arg2 ... [color=purple]

  Required Arguments:

    function: The symbol name for the function you want probed, this is the
              minimum required for an entry, it will show up as the function
              name with no arguments.

        example: _cpu_up:

  Optional Arguments:

    format: The format to display the data on the timeline in. Use braces to
            enclose the arg names.

        example: CPU_ON[{cpu}]

    color: The color of the entry block in the timeline. The default color is
           transparent, so the entry shares the phase color. The color is an
           html color string, either a word, or an RGB.

        example: [color=#CC00CC]

    arglist: A list of arguments from registers/stack addresses. See URL:
             https://www.kernel.org/doc/Documentation/trace/kprobetrace.txt

        example: cpu=%di:s32

 Here is a full example entry. It displays cpu resume calls in the timeline
 in orange. They will appear as CPU_ON[0], CPU_ON[1], etc.

  [timeline_functions_x86_64]
  _cpu_up: CPU_ON[{cpu}] cpu=%di:s32 [color=orange]


Adding or Editing Dev Mode Timeline Source Functions
____________________________________________________

 In dev mode, the tool uses an array of function names to monitor source
 execution within the timeline entries.

 The function calls are displayed inside the main device/call blocks in the
 timeline. However, if a function call is not within a main timeline event,
 it will spawn an entirely new event named after the caller's kernel thread.
 These asynchronous kernel threads will populate in a separate section
 beneath the main device/call section.

 The tool has a set of hard coded calls which focus on the most common use
 cases: msleep, udelay, schedule_timeout, mutex_lock_slowpath, etc. These are
 the functions that add a hardcoded time delay to the suspend/resume path.
 The tool also includes some common functions native to important
 subsystems: ata, i915, and ACPI, etc.

 It is possible to add new function calls to the dev timeline by adding them
 to the config. It's also possible to copy the internal dev timeline
 functions into the config so that you can override and edit them. Place them
 in the dev_timeline_functions_ARCH section with the name of your architecture
 appended. i.e. for x86_64: [dev_timeline_functions_x86_64]

 Use the override-dev-timeline-functions option if you only want to use your
 custom calls, or leave it false to append them to the internal ones.

 The format is the same as the timeline_functions_x86_64 section. It's a
 list of functions (set using kprobes) which use both symbol data and function
 arg data. The args are pulled directly from the stack using this
 architecture's registers and stack formatting. Each entry can include up
 to four pieces of info: The function name, a format string, an argument list,
 and a color. But only the function name is required.

 For a full example config, see config/custom-timeline-functions.cfg. It pulls
 all the internal dev timeline functions into the config and allows you to edit
 them.

 Here is a full example entry. It displays the ATA port reset calls as
 ataN_port_reset in the timeline. This is where most of the SATA disk resume
 time goes, so it can be helpful to see the low level call.

  [dev_timeline_functions_x86_64]
  ata_eh_recover: ata{port}_port_reset port=+36(%di):s32 [color=#CC00CC]


Verifying your custom functions
_______________________________

 Once you have a set of functions (kprobes) defined, it can be useful to
 perform a quick check to see if you formatted them correctly and if the system
 actually supports them. To do this, run the tool with your config file
 and the -status option. The tool will go through all the kprobes (both
 custom and internal if you haven't overridden them) and actually attempts
 to set them in ftrace. It will then print out success or fail for you.

 Note that kprobes which don't actually exist in the kernel won't stop the
 tool, they just wont show up.

 For example:

 sudo ./sleepgraph.py -config config/custom-timeline-functions.cfg -status
 Checking this system (myhostname)...
    have root access: YES
    is sysfs mounted: YES
    is "mem" a valid power mode: YES
    is ftrace supported: YES
    are kprobes supported: YES
    timeline data source: FTRACE (all trace events found)
    is rtcwake supported: YES
    verifying timeline kprobes work:
         _cpu_down: YES
         _cpu_up: YES
         acpi_pm_finish: YES
         acpi_pm_prepare: YES
         freeze_kernel_threads: YES
         freeze_processes: YES
         sys_sync: YES
         thaw_processes: YES
    verifying dev kprobes work:
         __const_udelay: YES
         __mutex_lock_slowpath: YES
         acpi_os_stall: YES
         acpi_ps_parse_aml: YES
         intel_opregion_init: NO
         intel_opregion_register: NO
         intel_opregion_setup: NO
         msleep: YES
         schedule_timeout: YES
         schedule_timeout_uninterruptible: YES
         usleep_range: YES


------------------------------------------------------------------
|           TESTING ON CONSUMER LINUX OPERATING SYSTEMS          |
------------------------------------------------------------------

Android
_______

 The easiest way to execute on an android device is to run the android.sh
 script on the device, then pull the ftrace log back to the host and run
 sleepgraph.py on it.

 Here are the steps:

 [download and install the tool on the device]

	host%> wget https://raw.githubusercontent.com/intel/pm-graph/master/tools/android.sh
	host%> adb connect 192.168.1.6
	host%> adb root
	# push the script to a writeable location
	host%> adb push android.sh /sdcard/

 [check whether the tool will run on your device]

	host%> adb shell
	dev%> cd /sdcard
	dev%> sh android.sh status
		host    : asus_t100
		kernel  : 3.14.0-i386-dirty
		modes   : freeze mem
		rtcwake : supported
		ftrace  : supported
		trace events {
		    suspend_resume: found
		    device_pm_callback_end: found
		    device_pm_callback_start: found
		}
	# the above is what you see on a system that's properly patched

 [execute the suspend]

	# NOTE: The suspend will only work if the screen isn't timed out,
	# so you have to press some keys first to wake it up b4 suspend)
	dev%> sh android.sh suspend mem
	------------------------------------
	Suspend/Resume timing test initiated
	------------------------------------
	hostname   : asus_t100
	kernel     : 3.14.0-i386-dirty
	mode       : mem
	ftrace out : /mnt/shell/emulated/0/ftrace.txt
	dmesg out  : /mnt/shell/emulated/0/dmesg.txt
	log file   : /mnt/shell/emulated/0/log.txt
	------------------------------------
	INITIALIZING FTRACE........DONE
	STARTING FTRACE
	SUSPEND START @ 21:24:02 (rtcwake in 10 seconds)
	<adb connection will now terminate>

 [retrieve the data from the device]

	# I find that you have to actually kill the adb process and
	# reconnect sometimes in order for the connection to work post-suspend
	host%> adb connect 192.168.1.6
	# (required) get the ftrace data, this is the most important piece
	host%> adb pull /sdcard/ftrace.txt
	# (optional) get the dmesg data, this is for debugging
	host%> adb pull /sdcard/dmesg.txt
	# (optional) get the log, which just lists some test times for comparison
	host%> adb pull /sdcard/log.txt

 [create an output html file using sleepgraph.py]

	host%> sleepgraph.py -ftrace ftrace.txt

 You should now have an output.html with the android data, enjoy!