Postman automation reminders

Also see "Postman to check Open API's are Running"

Fire Postman collections on demand using curl

A monitor is already setup: I need the postman monitor id and an API key

Run local postman collection using Newman via Powershell (call from CI pipelines or a short-cut on the desktop)

Postman Monitor for Continuous Monitoring and Alerting in MS Teams

Overview: Pretty much every tester and developer loves postman. And that is because it makes our lives easier and it just plain awesome.  Postman is bringing out tons of new features and I was playing around today looking how I could do continuous monitoring with my postman collections.

Thoughts & Playing:

I have a postman collection that runs 8 requests and does 14 asserts.  The first request gets a new OAuth token using AAD login.  Then I do a series of requests and I do an assert to ensure I am getting a 200 response code and that the response time is less than 3 seconds on each call.  I can run the collection locally.  Level 100 API verification looks good.

Tip: Monitor API logs using "The RED Method":

• ·     Rate (the number of requests per time); 
•       Errors; and
•       Duration. 

In the past, I have taken this collection and run it as a shortcut on my desktop using Powershell with the Postman CLI to display me the results.  Makes my life easier.

I then added Elgato stream deck so I can run the monitor with a single button push (more me playing than real value).  I'd say I'm at level 200 in continuous monitoring capability.

Next, I setup a monitor on the collection, and this allows me to login and view the dashboard and trace, great stuff, and I get an email if anything goes wrong as an alert.  So now I'm getting serious about monitoring and alerting on my API's.  Level 300 is approaching.

Postman monitoring has integration for MS Teams, and Slack.  It also can send logs to Data Dog and New Relic but now Application Insights (recon this will come soon).  I setup a channel in teams to have a webhook, and I can send in the results using Postman but it's way easier to use the integration on the Monitor to push the result of each run or automatically after 3 failures.

Summary:  This Postman monitoring allows me to send detailed API requests at different intervals so I'm thinking for production: 

• 5 min for health and basic check (look for performance and service slowdown or failure; add alerts but don't over alert so use teams except if service breaks then Teams groups),
• Hourly, check key functionality/API's including CRUD operations and clean up (ensure the service is operating for most key endpoints), and
• Daily, in the early hours run a full regression API set of tests, and clean up afterwards (Support/help desk need to review each day).

Don't over alert, let me say that again don't over alert.  Alerting is like water, you definitely a little but floods are not great.  So with Teams & Slack, it's easy to push results and issues into a channel so key people are aware, and it gives a much better experience than email alerting.

I like the idea of using Postman as it's infrastructure is separate as I generally use the Azure/MS stack including Application insights. 

What Next:  I'd like to figure out how to push results into my logs for reporting off a single source.  I could embed the postman monitoring into iFrames but I'd probably use an Azure logic apps Azure function to listen for the Postman POST, then I can format adaptive cards for Teams, and outlook, easily integrate Twilio for SMS or maybe What's app.  From the logic app i can use a Application Insights SDK to add Tracing.  

Combining with Correlation Id's and App Insights, I can see issues, have them summarised, get the right level of alerting, trace specific issues quickly.  Ideally we capture issues before customers report them. and if a customer reports and issue it can be 100% traced, remediated and fixed for all customers quickly.  Changes to API's and compatibility is also a nice benefit of this approach.

  

Postman to verify OpenAPI's are running

Problem:  Our teams rely on a 3rd party API for a new project being delivered, the API's are in a state of change and are constantly up and down making life tough for the teams replying on the API.

Hypothesis:  I need a quick way to check the API's to see if they are all working in dev, and test.  I have two postman collections for the REST API's.  If i combine them and check the key API's using postman I can save myself and other time as I'll know the current state of the API's.

Solution: Create a site collection that does the API verification, you can make it more complex with data and variables.

Problem:  I can open Postman and run the test which takes a few minutes.  We need to do this quicker.

Hypothesis: I'd like to be able to run the tests quickly on demand.  Use postman CLI and Powershell to run the collection and display the result.

Solution: 

1) Add the Postman CLI to my machine:

PS> powershell.exe -NoProfile -InputFormat None -ExecutionPolicy AllSigned -Command "[System.Net.ServicePointManager]::SecurityProtocol = 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://dl-cli.pstmn.io/install/win64.ps1'))"

2) In postman generate an API Key for the Collection > Run Collection > Automate runs via CLI > Generate the API Key > Copy the generated code

3) Run the code in PS to verify it works correctly.

4) Copy the PS code into a newly Created ps1 file on your local machine, I added a read line so I can see the result.

5) Run the API.ps1 file and verify the result

6) Setup a desktop short-cut to run and see the result.  Right click the API.ps1 file and create a shortcut on your desktop.  Right click and amend the target and amend the target value:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy Bypass -File C:\Users\PaulBeck\Downloads\Projects\PoC\Postman\API.ps1

7) Save and run the shortcut to verify.

Problem:  Monitor and alert DTAP API's are working and performance

Resolution: I want to monitor that my endpoints specified in my Postman collection in Dev, UAT et al. are working, can be more than 1 endpoint using Postman Monitor.

Next steps: Add to automated DevOps processes, using Newman.

App Insights Overview for SaaS logging and tracing

Overview: App Insights provides a standalone infrastructure for logging and tracing.  It is tightly coupled with Azure services, including PaaS.  This allows for consistent, scalable logging.  App Insights now stores logs in Azure Log Analytics; these are all under the umbrella of Azure Monitor. 

On a SaaS solution, I am looking for App Insights to log any errors and have the ability to log trace information.  I want a unique correlationId (to allow for distributed tracing) on the front end if there is an error, so support can identify the exact issue/transactions.  A unique correlationId in the HTTP header allows identifying a transaction, which is useful for tracing and performance monitoring.  Using the App Insights SDK's and implementing a standard logging module is a good idea.  Two common areas need to be called out to ensure the ability to trace transactions:

1. SPA's (Requirement to generate a unique operation/correlationId per operation not per pageview), and
2. Long-running operations, such as timer jobs or service bus calls.

Support & DevOps:

Having a correlationId allows the first line to log the correlationId and quickly follow the request without asking for replication steps.  This context tracing approach is common in newer applications. Third-line support has full traceability of an issue to support who can empirically see the performance components broken down using the correlationId in the header.

Key APIs can be continuously monitored for errors and performance slowdowns, and alerts can be configured based on these metrics. 

Building a first-line support tool that displays errors in a hierarchy, includes help scripts, and integrates a knowledge base is a good option for streamlining support.

App Insights has live monitoring, and the Kusto query language is helpful for monitoring specific queries.

Summary Report for Support

// I'm sure there are nicer ways to write/improve my Kusto, so pls let me let me know where the code can be improved

let dayminus0 = datetime(now);

let dayminus1 = ago(24h);

let dayminus2 = ago(48h);

let result0 = requests

    | where timestamp > dayminus1 and timestamp < dayminus0

    | summarize requestCount=sum(itemCount), avgDuration=avg(duration) by performanceBucket

    | where performanceBucket == "15sec-30sec" or performanceBucket == "7sec-15sec"

        or performanceBucket == "30sec-1-min" or performanceBucket == "1min-2min";

let dayminus1a = ago(24h);

let dayminus2a = ago(48h);

let result1 = requests

    | where timestamp > dayminus2a and timestamp < dayminus1a

    | summarize requestCount1=sum(itemCount), avgDuration1=avg(duration) by performanceBucket

    | where performanceBucket == "15sec-30sec" or performanceBucket == "7sec-15sec"

        or performanceBucket == "30sec-1-min" or performanceBucket == "1min-2min";

let dayminus1b = ago(2d);

let dayminus2b = ago(3d);

let result2 = requests

    | where timestamp > dayminus2b and timestamp < dayminus1b

    | summarize requestCount2=sum(itemCount), avgDuration2=avg(duration) by performanceBucket

    | where performanceBucket == "15sec-30sec" or performanceBucket == "7sec-15sec"

        or performanceBucket == "30sec-1-min" or performanceBucket == "1min-2min";

let resultTemp = result0

    | join kind=inner result1 on performanceBucket 

    | project performanceBucket, ['Today'] = avgDuration, ['Yesterday'] = avgDuration1;

let resultTemp2 = resultTemp;

resultTemp2

| join kind=inner result2 on performanceBucket 

| project

    performanceBucket,

    ['1) Today']= (round(['Today'], -2) / 1000),

    ['2) Yesterday'] = (round(['Yesterday'], -2) / 1000),

    ['3) Two Day ago'] = (round(avgDuration2, -2) / 1000) 

| render columnchart

    with (

    kind=unstacked,

    ytitle="Seconds Taken",

    xtitle="Performance Group",

    title="Ensure the 'Today' bar is not significantly higher than pervious days");

Monitoring:  Azure dashboards are great for monitoring application health and performance.  Easy to customise, make unique dashboards and security is easy to control.  sentry.io monitors API's, I have not used it.  I like all the Azure stuff coming out for testing and I feel continuously running Postman collections and reporting to App Insights is the best way to go.  Azure Dashboards can be limiting, Azure Grafana can be a great alternative/enhancement.  Check out Azure Managed Grafana.

source cloudiqtech

Alerting: I all too often see an overuse of alerting, resulting in recipients ignoring a plethora of emails.  I believe in minimising alerts, primarily via email and SMS.  For me, I like to create a dedicated channel for alerting that includes all DevOps members and either notify via a Teams card, or even easier, email the channel.  This can be broken down further but to start I create a channel for alerting for each DTAP environment.

Note: The default channel setup only allows members of the Teams channel to send email, so the alerts from Azure Monitor using rules won't be accepted.  On the channel, and admin needs to go to the "advance settings" and change the option from "Only members of this Team" and change it the setting to "Anyone can send".

Options: There are excellent logging services, so my default is Azure Monitor.  All leading vendors support Open Telemetry.  The leading players in Application & API observability and monitoring include: 

• Open Telemetry records traces, metrics, and logs from multiple distributed systems and is open source.
• Fluent Bit Log processor and forwarder.  Used to collecting, filtering, and shipping logs.  More restrictive than OpenTelemetry but lightweight.  Fluent Bit logs are often streamed or forwarded to OpenTelemetry loggers. Focuses on collecting, filtering, and shipping logs for smaller systems or devices.

• Microsoft: Azure Monitor includes Application Insights & Azure Log Analytics
• Dynatrace (really good if you use multicloud) or Dynatrace AWS CloudWatch,  Dynatrace - Saas offering is on AWS.  Can be on-prem.  One Agent is deployed on the Compute i.e. VM, Kubernetes.  Can import logs from other SIEMs or Azure Monitor, so you can eventually get Azure service logs such as App Service or Service Bus.  Does Full stack and includes code-level and applications and infrastructure monitoring, and can show User monitoring.  Dynatrace offers scalable APIs running on Kubernetes.  "Davis" is the AI engine used to identify problems.  Alerting is solid.  

High-level Architecture

Dynatrace Admin Monitoring

• AWS: Amazon CloudWatch Synthetics
• AppDynamics,
• Datadog (excellent),
• New Relic,
• SolarWinds (excellent)

SolarWinds admin UI from circa 2013/2014 

Dynatrace

SaaS Azure Testing Thoughts

 Tooling:

1. API Automation - Postman, Newman
2. UI Automation - Selenium
3. IDE - Visual Studio 2019
4. Test Organization - Azure DevOps Test Plan
5. CI/CD - Azure DevOps

Code reviews:

Code review is used as a verification technique to ensure that each unit is coded as per standards and expected business logic and inline with coding standards and best practices.  Automate code review built into Azure Pipelines should include:  

• WhiteSource Bolt - Scan packages for vulnerabilities.
• SonarQube - Static Analysis, 
• Blackduck - Open-Source Scanning (OSS) tool.  Used to look for license risks and unused references.
• Checkmarx - Static Application Security Testing (SAST) tool benefits include: Detect security vulnerabilities, Improve developer practices, and reports on code ownership.  Static code anaylsis.  VeraCode is a competitor product.
• BugSuite

Code should pass OWASP (Open Web Application Security Project) shows the most common code vulnerabilities.  OWASP ASVS (Application Security Verification Standard) - framework for controls when building applications to cover functional and NFR's for web applications.

Unit testing:

Unit tests are written to ensure every unit of code is working as expected, and to prevent a defect from going to the next level on all C# code.  Xunit and Moq are the tools to be used for unit testing using the standard Arrange > Act > Assert pattern.

As long as Unit test coverage is high and of a good standard, I don't mind if the tests are written before the code (TDD) or as most developers tend to do the tests after the code is written.

API testing:

All API must use Postman collections and Environments for local testing.  The tests need to cover all API's dealing with authentication, authorisation, checking status codes, body responses, headers, data persistence, and post test clean-up.  Use Newman to integrate postman tests into Azure pipelines:

https://www.npmjs.com/package/newman-reporter-htmlextra

Selenium testing:

Code for UI must be automated where possible.

SonarQube: "automatic code review tool to detect bugs, vulnerabilities, and code smells in your code" SonarQube documentation

Code Smells:  Bloaters, OO abusers, ....

Checkmax detects potential security issues

Disposable email addresses: You often need to test login/account creation and it's useful to have temporary disposable email addresses:

• YOPmail - Disposable Email Address
• Home - Mailinator