YuWebdesign



API, HTTP, and Networking Interview Questions

By YuwebDesign


General Networking Interview Questions

Network is a set of interconnected devices that communicate with each other using a physical transmission medium.

A Computer Network is a group of computers connected with each other to communicate and share information and resources like hardware, data, and software across each other.

A link between client and the server is the one that creates the network.

In a network, nodes are used to connect two or more networks.

Basic criteria for building a computer network
  1. At least two objects that communicate together on a server-client communication basis.

    Server
    Computer that manages that data or any other shared resources/Services in the local network.

    Client
    Computer that used by the user to interact in the network.

  2. Communication medium
  3. Devices can communicate with different types of communication mediums (Lab cables, Wireless).

  4. Network interface card (NIC) that can send/receive data.
  5. Supported Operating system
    (MacOS, WinOS…).

Network Examples:

  1. Social networks (Facebook, Google, Twitter…).
  2. Home networks (2-10 computers) that share data.
  3. Academic Networks (sharing data and academic information).
  4. The biggest network of all is the “Internet”.
  5. Company networks, which allow remote access, centralized administration and much more.

Networks are Easy to use
Networking is easy, anyone can use social networks or insert credentials for authentication against the enterprise servers, as a software testers, we need to be familiar with the procedures and how it’s really work.

Networks are Reliable
Since the attackers will always be part of a fast growth technology, not all networks are safe to use. However high percentage of big networks is reliable for the average user and safe for use.

Networks are efficient
The efficiency of networks is well known, just think about the internet and how it affects us, the access to information, the ability to share our data and what we think.

  1. When two or more computers are connected directly by an optical fiber or any other cable,
    a node is a point where a connection established.
  2. It is a network component used to send, receive and forward the electronic information.
  3. A device connected to a network is also a node.

E.g, there are five nodes on the network connecting 2 computers, 2 printers, and a server.

Network Topology is a physical layout of the computer network.

It defines how the computers, devices, cables etc. are connected to each other.

LAN (Local Area Network)

A collection of standalone computers that transfer and share information with each other over a communication medium.

Local Area Networks (LAN) are usually used when the network objects reside in a similar geographic location (in close proximity), e.g.:

  1. Same university
  2. Same building
  3. Same office
  4. Home network
LAN networks scale
LAN Size Number of objects
(Computers, printers, servers)
Small Average of 10-100 objects
Medium Average of 100-500 objects
Large Average of 500+ objects

In addition, LANs networks are usually less expensive than WANs networks, the reason is the close proximity of the network objects.

To allow communicationб IT team can use a basic Hardware such has modems, wireless Modems, Hubs and regular communication cables.

WAN (Wide Area Network)

Think about LANs and multiply it.
Wide Area Networks is a combination of multiple LANs located in a different geographical location (in other words, we have few LANs located under a different physical location).

The connection between the LANs accomplished with a specific set of Hardware devices that cost much more that they once used to build the LAN network. Among those devices we can find Satellite links, communication lines (Phone lines, dedicated ISP lines and more) that allow the communication between different LANs.

The real power of WAN networks can be explained in a very simple way.
The Internet is a complex system built from massive amount of linked networks that exists worldwide and allow the users to communicate with other people (Remote login, Mail systems, Facebook, websites and more).

VPN (Virtual Private Network)

VPN network is a basic way to connect to your local network (LAN) from anywhere you desire.

VPN uses a technique called “Tunneling” used to send and receive any data across the public and private networks.

A VPN connection is usually established by creating a Virtual “Point-to-Point” using a dedicated connection (Tunneling, Encryption).

Using VPN you can access to your company resources with the same security as you work on your company personal computer. In addition, you can access the company services (Mail, dedicated application Etc.).

Wireless Local Area Networks (WLANs)

WLAN networks use the radio frequency (RF) technology to communicate and exchange data over the air (No need to use physical cables between the LANs).

WLAN have a great advantage over LANs because they allow the users to access the data without the need to connect a physical cable to the local LAN, which can save time and allow users to use different devices such as phones, tablets and any other objects that can work with RF technology.

This freedom does not come without a price: to access such devices users should authenticate against the LAN with a specific authentication protocol, the two main protocols that support such networks are WEP and WPA.

  1. Circuit switching uses physical path for transmission of data.
  2. Here initially a physical connection is established between source and destination & then data is sent.
  3. In packet switching data is broken into small parts called packets.
  4. Numbered packets may take any shortest route for transmission.
  5. In packet switching virtual connection is made between source and destination.

  1. Vertical Redundancy Check (VRC)
  2. Longitudinal Redundancy Check (VRC)
  3. Cyclic Redundancy Check (VRC)
  4. Checksum

  1. Bit stuffing means adding one extra 0 whenever five consecutive 1s follow a 0.
  2. This is done so that the pattern 0111110 is not mistaken for a flag.

Bandwidth and latency describe the performance of a network.

Bandwidth/ Throughput

The number of bits which can be transmitted over the network in a specified time.

Latency

The time taken by a message to travel from one side to another.

  1. Electromagnetic Transmission

    The electromagnetic waves can be broadcast efficiently and received by a receiver some distance away.

  2. Radio Transmission

    Radio waves are easy to generate, can travel long distances, and can penetrate buildings easily, so they are widely used for communication. These waves can travel in all directions.

  3. Microwave Transmission

    Microwaves travel in a straight line, so if the towers are too far apart, the earth will get in the way. Thus, repeaters are needed periodically. The higher the towers are, the farther apart they can be. Unlike radio waves at lower frequencies, microwaves do not pass through buildings well.

  4. Infrared Transmission

    It is used for short-range communication. The remote controls used for televisions, VCRs, and stereos, use infrared communication. They are relatively directional and can not pass through solid objects.

  5. Light Transmission
    These types of wave can easily affected by wind and temperature changes. They normally work well on sunny days.

Signals need transmission media for travel from one point to another. Transmission media are not perfect because it provides resistance.
The imperfection causes signal impairment.
This means that the signal at the beginning of the medium is not the same as the signal at the end of the medium.
What is sent is not what is received.

There are three causes of impairment:

  1. Attenuation

    When a signal travels through a medium, it loses some of its energy due to resistance of the medium. This loss of energy is called Attenuation. This is the reason why a wire carrying electric signals gets warm. Some of the electrical energy in the signal is converted to heat. To overcome for this problem, amplifiers are used to amplify the signal.

    Unit of the decibel is used to find out if a signal has lost or gained strength. The decibel (dB) measures the relative strengths of two signals or one signal at two different points. If decibel is negative then a signal is attenuated and if positive signal is amplified.

  2. Distortion

    When the signal travels through the medium from one point to another it may chance to change the form or shape of signal. It is called distortion.

    Distortion can occur in a composite signal, made up of different frequencies. Each signal component has its own propagation speed through a travelling medium and, therefore, its own delay in reaching at the final destination. Means signal components at the receiver have phases different from what they had at the sender.

  3. Noise

    The third cause of impairment is Noise. Following types of noise are possible.

    1. Thermal noise

      Thermal noise is produced due to the random motion of electrons in a wire which creates an extra signal not originally sent by the transmitter.

    2. Induced noise

      Induced noise comes from sources such as motors and other electronic appliances.
      These devices act as a sending antenna, and the transmission medium acts as the receiving antenna.

    3. Crosstalk noise

      Crosstalk is the effect of one wire on the other.

    4. Impulse noise
      Impulse noise comes from power lines, lightning etc.

    These noise may corrupt the signal.

RTT stands for round-trip time. It is the time taken by a packet to travel from client to server and then back to the client.

In the process of fetching a web page from a server the HTTP request/response takes 1 RTTs.

By default the http connection will be persistent connection. Hence it will take only 1 RTT to fetch a webpage from a server.

Logon scripts are, surprisingly enough, scripts that run at logon time. These are used most times to allow for the continued access to share and device mapping as well as forcing updates and configuration changes. In this way, it allows for one-step modifications if servers get changed, shares get renamed, or printers get switched out for example.

An undeliverable datagram is discarded and an ICMP error message is sent to the source host.

When trying to communicate with systems on the inside of a secured network, it can be very difficult to do so from the outside- and with good reason. Therefore, the use of a port forwarding table within the router itself or other connection management device, can allow for specific traffic to be automatically forwarded on to a particular destination. For example, if you had a web server running on your network and you wanted access to be granted to it from the outside, you would setup port forwarding to port 80 on the server in question. This would mean that anyone putting in your IP address in a web browser would be connected up to the server’s website immediately. Please note, this is usually not recommended to allow access to a server from the outside directly into your network.

Sticky ports are one of the network admin’s best friends and worst headaches. They allow you to set up your network so that each port on a switch only permits one (or a number that you specify) computer to connect on that port by locking it to a particular MAC address. If any other computer plugs into that port, the port shuts down and you receive a call that they can’t connect anymore. If you were the one that originally ran all the network connections then this isn’t a big issue, and likewise if it is a predictable pattern then it also isn’t an issue. However if you’re working in a hand-me-down network where chaos is the norm then you might end up spending a while toning out exactly what they are connecting to.

Port Numbers
HTTP can use a number of different carrier protocols to go from system to system, the primary protocol and port used is TCP port 80.
HTTPS HTTPS traffic goes over TCP port 443.
FTP FTP uses TCP ports 20 and 21.
SSH SSH runs over TCP port 22.

See list of TCP and UDP port numbers on wikipedia >>>

Being able to ping out to a server and see if its responding is a great way to troubleshoot connectivity issues. But what if you’re not able to ping ANY server? Does that mean that your entire network is down? Does it mean that your network cable needs to be replaced? Does it mean that your network card is going bad? Or could it possibly be that sunspots, magnets, aliens and the Men In Black are all conspiring against you? The answers to these questions could be very difficult, but at the very least you can rule out if your network card is going bad. 127.0.0.1 is the loopback connection on your network interface card (NIC)- pinging this address will see if it is responding. If the ping is successful, then the hardware is good. If it isn’t, then you might have some maintenance in your future. 127.0.0.1 and localhost mean the same thing as far as most functions are concerned, however be careful when using them in situations like web programming as browsers can treat them very differently.

There are two types of client/server programs:

  1. Used directly by the user,
    e.g., e-mail
  2. Supports other application programs,
    e.g., DNS that is used by other programs (e.g., to find the IP address of an e-mail recipient).

DNS (Domain Name System) could be compared to the Internet’s phone book.

DNS makes it possible to remember something like “cnn.com” instead of “157.166.226.26”.

IP addresses change all the time (less for mega-level servers).
DNS makes it possible to map human friendly names to new IP addresses under the hood.

Standard phone book allows you to look for name of the person or business, and then it will show you the number for that person. DNS servers do exactly the same but with updates on a daily or hourly basis.

The tiered nature of DNS also makes it possible to have repeat queries responded to very quickly, although it may take a few moments to discover a brand new address that you haven’t visited before.

Your computer will first talk to your local DNS server (likely your home router) to see if it knows where it is. If it doesn’t know, it will talk to your ISP’s DNS server and ask it if it knows. If the ISP doesn’t know, it will keep going up the chain asking questions until it reaches one of the 13 Root DNS Servers. The responding DNS server will send the appropriate address back down the pipe, caching it in each location as it does so to make any repeat requests much faster.

ARP, or Address Resolution Protocol can be likened to DNS for MAC Addresses. Standard DNS allows for the mapping of human-friendly URLs to IP addresses, while ARP allows for the mapping of IP addresses to MAC addresses. In this way it lets systems go from a regular domain name down to the actual piece of hardware it resides upon.

ARP Protocol Packet Structure, Data Unit Structure

The fields of the packet format of ARP

:

  1. Hardware type: It is a 16-bit field . Each LAN has been assigned an integer based on its type. For example, Ethernet is given the type 1.
  2. Protocol type: This is a 16-bit field defining the protocol. For example, the value of this field for IPv4 protocol is 0800.
  3. Hardware length: This is an 8-bit field defining the length of the physical address in bytes. For example, for Ethernet the value is 6.
  4. Protocol length: This is an 8-bit field defining the length of the logical address in bytes. For example, for IPv4 protocol the value is 4.
  5. Operation: This is a 16-bitfield defining the type of packet. Two packet types are defined: ARP request (1), ARP reply (2).
  6. Sender hardware address: This is a variable-length field defining the physical address of the sender. For example, for Ethernet this field is 6 bytes long.
  7. Sender protocol address: This is a variable-length field defining the logical (for example, IP) address of the sender. For the IP protocol, this field is 4 bytes long.
  8. Target hardware address: This is a variable-length field defining the physical address of the target. For example, for Ethernet this field is 6 bytes long.
  9. Target protocol address: This is a variable-length field defining the logical (for example, IP) address of the target. For IPv4 protocol, this field is 4 bytes long.

A workgroup is a collection of systems each with their own rules and local user logins tied to that particular system. A Domain is a collection of systems with a centralized authentication server that tells them what the rules are. While workgroups work effectively in small numbers, once you pass a relatively low threshold (usually anything more than say 5 systems), it becomes increasingly difficult to manage permissions and sharing effectively. To put this another way, a workgroup is very similar to a P2P network- each member is its own island and chooses what it decides to share with the rest of the network. Domains on the other hand are much more like a standard client/server relationship- the individual members of the domain connect to a central server which handles the heavy lifting and standardization of sharing and access permissions.

In a hierarchical name space, each name is combination of several parts as:

Type of the organization.
Name of an organization.
Departments in the organization.

A central authority can assign the part of the name that defines the nature and name of the organization. The other things of the name the organization provides itself. The organization can add suffixes (or prefixes) to the name to define its host.

The company need not worry about the prefix if it is same as that of another organization because, even if part of an address is the same, the whole address is different.

Example:

Suppose central authority has given names ABC.com and XYZ.com to two separate organization but these organization can add prefix as career so the name becomes career.XYZ.com and career.ABC.com. The names are unique without the need for assignment by a central authority. The central authority controls only part of the name, not the whole.

Hierarchical Namespace

In hierarchical name space, names are defined in an inverted-tree structure with the root at the top. The tree can have only 128 levels: level 0 (root) to level 127.

Each node in the tree has a label, which is a string with a maximum of 63 characters.
The root label is a null string (empty string). In this tree, all the labels have different name, which guarantee the uniqueness of the domain names.

A full domain name is a sequence of labels separated by dots. The domain names are always read from the node up to the root.

Fully Qualified Domain Name (FQDN)
If a label is terminated by a null string or empty string, it is called a fully qualified domain name.

FQDN contains both the hostname and a domain name.
FQDN uniquely identifies a host within a DNS hierarchy.
Partially Qualified Domain Name (PQDN)

If a label is not terminated by a null string, it is called a partially qualified domain name. A PQDN starts from a node, but it does not end with the root.

When you’re working in Active Directory, you see a tree-type structure going down through various organizational units (OU’s). The easiest way to explain this is to run through a hypothetical example.

Say that we had a location reporting for CNN that dealt with nothing but the Detroit Lions. So we would setup a location with a single domain, and computers for each of our users. This would mean starting at the bottom: OU’s containing the users, groups and computers are at the lowest level of this structure. A Domain is a collection of these OU’s as well as the policies and other rules governing them. So we could call this domain ‘CNNDetroitLions”. A single domain can cover a wide area and include multiple physical sites, but sometimes you need to go bigger.

A tree is a collection of domains bundled together by a common domain trunk, rules, and structure. If CNN decided to combine all of its football team sites together in a common group, so that its football sports reporters could go from one location to the next without a lot of problems, then that would be a tree. So then our domain could be joined up into a tree called ‘football’, and then the domain would be ‘CNNDetroitLions.football’ while another site could be called ‘CNNChicagoBears.football’.

Sometimes you still need to go bigger still, where a collection of trees is bundled together into a Forest. Say CNN saw that this was working great and wanted to bring together all of its reporters under a single unit- any reporter could login to any CNN controlled site and call this Forest ‘cnn.com’ So then our domain would become ‘CNNDetroitLions.football.cnn.com’ with another member of this same Forest could be called ‘CNNNewYorkYankees.baseball.cnn.com’, while yet another member could be ‘CNNLasVegas.poker.cnn.com’. Typically the larger an organization, the more complicated it becomes to administer, and when you get to something as large as this it becomes exponentially more difficult to police.

As per Microsoft:

“A domain local group is a security or distribution group that can contain universal groups, global groups, other domain local groups from its own domain, and accounts from any domain in the forest. You can give domain local security groups rights and permissions on resources that reside only in the same domain where the domain local group is located.

A global group is a group that can be used in its own domain, in member servers and in workstations of the domain, and in trusting domains. In all those locations, you can give a global group rights and permissions and the global group can become a member of local groups. However, a global group can contain user accounts that are only from its own domain.

A universal group is a security or distribution group that contains users, groups, and computers from any domain in its forest as members. You can give universal security groups rights and permissions on resources in any domain in the forest. Universal groups are not supported.”

Tracert or traceroute depending on the operating system allows you to see exactly what routers you touch as you move along the chain of connections to your final destination. If you end up with a problem where you can’t connect or can’t ping your final destination, a tracert can help in that regard as you can tell exactly where the chain of connections stop. With this information, you can contact the correct people- whether it be your own firewall, your ISP, your destination’s ISP or somewhere in the middle. Tracert, like ping, uses the ICMP protocol but also has the ability to use the first step of the TCP three-way handshake to send out SYN requests for a response.

A subnet mask tells the network how big it is. When an address is inside the mask, it will be handled internally as a part of the local network. When it is outside, it will be handled differently as it is not part of the local network. The proper use and calculation of a subnet mask can be a great benefit when designing a network as well as for gauging future growth.

  1. Text file that resolves NetBIOS names to IP addresses.
  2. Can be compared with the HOSTS file which is a static method of resolving domain names into IP addresses.
  3. LMHOSTS files must be manually configured and updated.

Boot to LAN is most often used when you are doing a fresh install on a system. What you would do is setup a network-based installer capable of network-booting via PXE. Boot to LAN enables this by allowing a pre-boot environment to look for a DHCP server and connect to the broadcasting network installation server. Environments that have very large numbers of systems more often than not have the capability of pushing out images via the network. This reduces the amount of hands-on time that is required on each system, and keeps the installs more consistent.

Networking Models

The P2P model in its simplest form is created when you have two or more computers connected in a basic network that allows them to share resources between each other (one computer can share any info without a third computer in the middle).

Simply put, in this model every computer can be used both as the Server or the client (not like the Server-Client model in which the client sends the request to the server and waits until the server fulfils the request) with equivalent responsibility for processing data.

This model is very friendly to end users, when a new user wants to add a computer to a P2P network, they need to follow few basic steps:

  1. Connect the computer to the physical network (P2P computers run the same communication protocols).
  2. Download the P2P supported software.
  3. Execute the software.
  4. After the software becomes active, the user can see the other computers belonging to the network by their IP address.
  5. A server that server a program will run forever until a failure occurs (the server doesn’t know when the clients will send a request, therefore he need to be open all the time).
  6. When the connection established with one of the other computers, the share of data can be created (Share Files, Access to resources, Shared printers, playing computer games Etc.).
  7. The basic network protocols that used to communicate between the network objects are User Datagram Protocol (UDP) and Transmission Control Protocol (TCP).
  8. Most of the application processing is done on the client side, the application services are provided by the remote server.

In this model, there is a structured relationship between the server and the clients.

The client computer is the one that requests the relevant services from the centralized server (connection established using LAN or WAN, Internet is a great example) that receive the request and reply with the needed results.

Basic Flow
  1. The client wants to access a specific service on a remote server.
  2. A communication established between the client-Server using the IP address and Port of the remote server.
  3. The request is responded or Declined by the server.
  4. Based on phase 3, the client closes the communication to the remote server.
In Client-Server Model
  1. The server is the one that provides the services, the client sends the request it.
  2. Local computers can communicate against other clients or different devices (printers are the most common).
  3. The architecture of this model embraced the separation between the servers and the clients.
  4. The servers can serve as both: servers and clients.
  5. If the centralized server is shut down or out of service, the clients cannot access the resources that this server holds.
Benefits of Client-Server Model

Central management
The central administration is probably the primary advantage over P2P networks.
In Client-Server model we can centralize the entire environment using one or more servers that control the entire clients connected to it.

Using the central administration, we can control the entire network in a several fields, e.g.:

  1. Single server that manages the network “Updates” from the internet (WSUS) server, and allow the IT to choose the relevant updates.
  2. Controlling the entire network objects (Servers, Computers, Clients and any other device that connected to the LAN).
  3. Enforce security over the network.
  4. Share resources from a secured location using policies that ensure that only the authorized clients can access and use this data.
  5. Reduce the need for multiple devices (think about 50 computers that can use a single shared printer).

Business Growth
Server-Client model will allow a major growth of your business, unlike P2P model relevant for a few clients (2-15).
The Client-Server model is built to support a thousand of users that can be managed by a few servers.

Security
Servers can be configured with highly advances modules that can manage the control and access of resources.
Client-Server model allows us to ensure that only the authorized clients will have access to the network services and resources.

Cost saving
Few examples that explains how this model can save a major amount of resources and money
(However, remember that server hardware will cost you more than a basic computer):

  1. Only few support staff representatives are required to maintain the network
    1. security policies.
    2. Servers.
    3. clients
    4. backup plans.
    5. services.
  2. Servers support virtualization that saves a major amount of money (Electricity, physical location, shared resources etc.).
  3. Reduce the number of licenses in a given network.

Remote Access
The servers allow remote access to the local network, which enables external users (Employees, Customers) to access the data located on the internal network.

Redundancy
The servers both Hardware and Software are designed to answer requests from clients with the fastest and redundant way as possible. The data stored in few different architectures that allow the IT team to recover any lost data quickly and efficiently.

Backup
Using dedicated servers applications, the company IT stuff can back up the server data and the clients computers with a specific backup plans, File servers for examples, will contain the company data on one central location, this allow us to backup this data on a single server and recover it on any case of lost data.

What is OSI?

When people talk about OSI, they are usually referring to what is formally called the Basic Reference Model (BRM) for Open Systems Interconnection (OSI). The BRM is currently maintained by the International Organization for Standards (ISO), although much of the initial drafting of the model and its early promulgation was done by the International Consultative Committee on Telegraphy and Telephony (CCITT), now known as the Telecommunications Standardization Sector of the International Telecommunications Union ( ITU-TSS or ITU-T).

In principle, the areas are distinct and of increasing generality; in practice, the boundaries between the layers are not always sharp.

The model draws a clear distinction between

  1. a service, something that an application program or a higher-level protocol uses,
  2. and the protocols themselves, which are sets of rules for providing services.

Read more about 7 Layers in the Basic Reference Model for Open Systems Interconnection >>>

There are two different, but related models, for the interconnection of computer networks:

OSI/ISO networking model TCP/IP approach
Headed by ISO Internet Engineering Task Force (IETF) and the Internet Architecture Board
Formality level The working groups concerned with OSI
are part of the formal international structure
for developing and maintaining standards
TCP/IP standards are somewhat less formal
The OSI movement started in the early 1990s.

Goal:

  1. providing standards for all computer hardware and software vendors
  2. for reducing multiplicity of interconnection and interface practices
  3. thus reducing the costs of designing and producing both hardware and software.

The U.S. Government, through the National Institute of Standards and Technology (NIST; formerly, the National Bureau of Standards), and many states have backed the OSI movement through contractual policies and legislation.

However, in many areas, the OSI movement has failed to win support for its formal standards, which have given way to standards imposed in the marketplace.

Although all major (and many minor) computer vendors now have OSI products, the OSI protocols were never widely implemented, and TCP/IP has become the first choice for multivendor networking.

Familiarity among college graduates (universities use TCP/IP, so once out of school, a graduate’s first choice when designing a system is to use TCP/IP)

Efficiency more-efficient implementation (less CPU time required, smaller programs)
Operational systems Availability for most operating systems
Speed Fast standardization and development cycle (usually using the Internet to facilitate communications) when a new requirement is identified
Ease of access Easier-to-access (and zero-cost) documentation and standards (they are all available on the Internet)
Price Lower-cost
Weaknesses OSI model has been very useful for talking about computer networks because of its emphasis on layering; but its protocols are very weak. TCP protocols are resounding successes, but the TCP conceptual model is weak.

TCP approach does little to distinguish the concepts of protocol, interface, and service.
These weaknesses are partially based on the ARPANET assumptions that network users are technical experts with great programming sophistication.

The design of protocols in the TCP/IP model of the Internet does not concern itself with strict hierarchical encapsulation and layering.
RFC 3439 contains a section entitled “Layering considered harmful”.
TCP/IP does recognize four broad layers of functionality which are derived from the operating scope of their contained protocols: the scope of the software application; the host-to-host transport path; the internetworking range; and the scope of the direct links to other nodes on the local network.

TCP/IP layer mapping to OSI/ISO

Despite using a different concept for layering than the OSI model,
TCP/IP layers are often compared with the OSI layering scheme.

  1. Layers 1-4 in general map between OSI and TCP/IP
    1. TCP/IP link layer corresponds to the OSI data link layer and may include similar functions as
      1. the physical layer
      2. some protocols of the OSI’s network layer.
  2. Layers 5-7 of OSI are mapped into a single Application Layer of TCP/IP
    1. The TCP Internet application layer maps to the OSI
      1. application layer (7)
      2. presentation layer (6)
      3. most of the session layer (5).
    2. gracefully closing a session:
      1. Session layer in OSI (5)
      2. Transport layer in TCP/IP(4)
    3. OSI Session layer (5)
      session checkpointing and recovery,
      which is not usually used in TCP/IP.
    4. The TCP/IP transport layer maps to
      1. graceful close function of the OSI session layer
      2. OSI transport layer
OSI
Layer
TCP/IP
Layer
Protocol
data
unit
(PDU)
Addressing Protocols
H
O
S
T

L
A
Y
E
R
S

7

APPLICATION


High-level APIs
Layer closest to the end user

Deals with the interface between a user (and other applications)
and the application that implements a communicating component.

Such application programs fall outside the scope of the OSI model.

Application-layer functions typically include

  1. identifying communication partners (identity and availability)
  2. determining resource availability
  3. synchronizing communication.

The most important distinction in the application layer:
application-entity vs. application.

  1. E.g, a reservation website might have two application-entities:
    1. one using HTTP to communicate with its users
    2. one for a remote database protocol to record reservations
  2. Neither of these protocols have anything to do with reservations.
    That logic is in the application itself.

The application layer per se has no means to determine the availability of resources in the network.

5

APPLICATION
(Process & Applications)


Provides applications services to users and programs
OSI:
Data

TCP/IP:
Messages

TCP/IP:
n/a
TCP/IP:
HTTP
HMTP


OSI:
FTAM
X.400
X.500
DAP
ROSE
RTSE
ACSE
CMIP
6
PRESENTATION
(sometimes called the Syntax Layer)
  1. Provides independence from data representation
    by translating between application and network formats.
  2. Transforms data into the form that the application accepts
  3. Formats data to be sent across a network
  4. Can include compression functions
  5. Negotiates the Transfer Syntax

Syntactic representation of data includes:

  1. agreement on character encoding (e.g., ASCII, extensions to ASCII, Unicode)
  2. data compression
  3. encryption/decryption.
  4. representations of graphics (e.g., files using the .PIC or .BMP formats)

The presentation layer establishes context between application-layer entities:

  1. the application-layer entities may use different syntax and semantics
    if the presentation service provides a mapping between them.
  2. If a mapping is available,
    presentation protocol data units are encapsulated into session protocol data units
    and passed down the protocol stack.
TCP/IP:
MIME
SSL
TLS
XDR


OSI:
ISO/IEC
8823X.226
ISO/IEC
9576-1X.236
5
SESSION


Controls the dialogues (connections) between computers:

  1. establishes
    (Сreating sessions
    when one application process
    requests access to another applications process
    (e.g., Microsoft Word importing a chart from Excel))
  2. manages
    (Managing communication sessions,
    i.e. continuous exchange of information
    in the form of multiple back-and-forth transmissions between two nodes.)
  3. terminates

the connections between the local and remote application.

Session Layer:

  1. Provides for
    1. full-duplex
    2. half-duplex
    3. or simplex

    operation

  2. Establishes procedures for
    1. checkpointing
    2. suspending
    3. restarting
    4. and terminating

    a session.

  3. commonly implemented explicitly in application environments that use remote procedure calls.
TCP/IP:
Sockets (session establishment in TCP / RTP / PPTP)


OSI:
ISO/IEC
8327X.225
ISO/IEC
9548-1X.235
4
TRANSPORT


Reliable transmission of data segments
between points on a network,
including

  1. segmentation
    creates segments out of the message received from the application layer.
    Segmentation is the process of dividing a long message into smaller messages.
  2. acknowledgement
    provides the acknowledgement of the successful data transmission and sends the next data if no errors occurred.
  3. and multiplexing.
  1. provides the functional and procedural means for data transfer between end systems
  2. controls the reliability of a given link through
    1. flow control
    2. segmentation/desegmentation
    3. and error control.

State- and connection-oriented protocols
This means that the transport layer can keep track of the segments and re-transmit those that fail delivery.

  1. OSI
    1. defines 5 classes of connection-mode transport protocols
      ranging from

      • class 0
        • known as TP0 providing the fewest features
        • no error recovery
        • designed for use on network layers that provide error-free connections
      • to class 4
        (TP4, designed for less reliable networks, similar to the Internet. Closest to TCP, although TCP contains functions, such as the graceful close, which OSI assigns to the session layer.).
    2. OSI TP connection-mode protocol classes provide expedited data and preservation of record
      boundaries.
  2. TCP/IP:
    Although not developed under the OSI Reference Model and not strictly conforming to the OSI definition of
    the transport layer, TCP and UDP of TCP/IP are commonly categorized as layer-4 protocols within OSI.
4
TRANSPORT


  1. Sorts out which client and server programs are supposed to get the data.
  2. Handles data-consistency functions, i.e., provides a reliable byte stream between two nodes on a
    network.
ISO:Segment,
Datagram

TCP/IP:
Segment

TCP/IP:
Port #’s
TCP/IP:
TCP
UDP
SCTP
DCCP


OSI:
ISO/IEC
8073TP0TP1TP2TP3TP4
(X.224)ISO/IEC
8602X.234
M
E
D
I
A

L
A
Y
E
R
S

3
NETWORK
(also called the Internet Layer in TCP/IP)


A network is a medium

  1. to which many nodes can be connected,
  2. every node has an address
  3. permits connected nodes transfer messages by
    1. providing message content and destination node address
    2. and letting the network find the way to deliver the message to the destination node,
      possibly routing it through intermediate nodes.

Internetwork is a collection of networks connected together through routers, the most famous of these being the Internet.

  1. Structuring and managing a multi-node network, including
    1. addressing
      (establishing paths for data between a pair of computers)
    2. routing
      (Allows different networks to communicate with each other through devices known as routers.
      handles any switching among alternative routes between the computers)
    3. and traffic control.
  2. Deals with definitions of how to break files (or messages) up into individual packets of data, in such a way that the packets can be transmitted and then reassembled.

    If the message is too large to be transmitted from one node to another on the data link layer between those
    nodes, the network may implement message delivery by

    1. splitting the message into several fragments at one node,
    2. sending the fragments independently,
    3. and reassembling the fragments at another node.
  3. provides the functional and procedural means
    of transferring variable length data sequences (called packets)
    between two individual nodes on the client-server basis
    connected in “different networks”.
  4. It may, but does not need to, report delivery errors.

    Message delivery at the network layer is not necessarily guaranteed to be reliable;
    a network layer protocol may provide reliable message delivery, but it need not do so.

  5. Provides network addressing and routing
    in a way to provide a common address space
    across multiple lower-level protocols.

    This makes possible the interconnection of networks that characterizes the Internet.

OSI:
Packet

TCP/IP:
Datagram

TCP/IP:
IP Address
TCP/IP:
IP
IPsec
ICMP
IGMP
OSPF
RIP


OSI:
ISO/IEC
8208X.25
(PLP)
ISO/IEC
8878X.223ISO/IEC
8473-1CLNP
X.233ISO/IEC
10589IS-IS
2
DATA LINK


  1. Reliable transmission of data frames (e.g., packets)
    between two nodes
    directly connected by a physical layer (physical link between network entities)
  2. Incorporation of error-correction coding into the data frames.
    It detects and possibly corrects errors that may occur in the physical layer.
  3. Defines the protocol to establish and terminate a connection between two physically connected devices.

    It also defines the protocol for flow control between them.

IEEE 802 divides the data link layer into two sublayers:

  1. Medium access control (MAC) layer:

    responsible for controlling how devices in a network gain access to a medium and permission to transmit data.

  2. Logical link control (LLC) layer:

    responsible for identifying and encapsulating network layer protocols,
    controls error checking and frame synchronization.

  1. The MAC and LLC layers of IEEE 802 networks such as 802.3 Ethernet, 802.11 Wi-Fi, and 802.15.4 ZigBee operate at the data link layer.
  2. The Point-to-Point Protocol (PPP) is a data link layer protocol that can operate over several different physical layers, such as synchronous and asynchronous serial lines.
  3. ITU-T G.hn standard, which provides high-speed local area networking over existing wires (power lines,
    phone lines and coaxial cables), includes a complete data link layer that provides both error correction and
    flow control by means of a selective-repeat sliding-window protocol.
2
DATA LINK
(also Network Interface of Network Access Layer)


  1. Defines a common way of interpreting signals so network devices can communicate.
  2. This layer contains whatever IP will run over, e.g., Ethernet, token-ring, and Fiber Distributed Digital Interface (FDDI) networks.
OSI:
Frame


TCP/IP:
Frames
MAC Address TCP/IP:
PPP
SBTV
SLIP

Individual network protocols, e.g., Ethernet and Wi-Fi, work at this level.

The Ethernet standards also define a protocol responsible for getting data to nodes on the same network or link.


OSI:
ISO/IEC
7666X.25
(LAPB)
Token
BusX.222ISO/IEC
8802-2LLC
(type 1 / 2)
1
PHYSICAL


Transmission and reception of unstructured raw data streams over a physical medium.

Deals with the physical (electrical and mechanical) aspects of transmitting data:

  1. voltage levels
  2. pin-connector design
  3. cable lengths
  4. grounding arrangements
  1. It converts the digital bits into electrical, radio, or optical signals.
  2. Layer specifications define characteristics

    such as voltage levels, the timing of voltage changes, physical data rates, maximum transmission distances, modulation scheme, channel access method and physical
    connectors.

    This includes the layout of pins, voltages, line impedance, cable specifications, signal timing and
    frequency for wireless devices.

  3. Bit rate control is done at the physical layer
    and may define transmission mode as
    simplex, half duplex, and full duplex.
  4. The components of a physical layer can be described in terms of a network topology.

    Bluetooth, Ethernet, and USB all have specifications for a physical layer.

1
PHYSICAL


  • Cabling, connectors and sending signals
  • Since TCP and IP, as protocols, deal with software rather than hardware, this layer is generally thought of as referring to all hardware under the Network Layer.
  • OSI:
    Symbol

    TCP/IP:
    10 Base T,
    802.11

    TCP/IP:
    N/a
    OSI:
    X.25
    (X.21bisEIA/TIA-
    232EIA/TIA-
    449EIA-
    530G.703)

    TCP/IP model serves primarily the protocols known as Transmission Control Protocol (TCP) and Internet Protocol (IP), or jointly, TCP/IP.

    The User Datagram Protocol (UDP) is also served by this model.

    TCP/IP 4-Layer Model combines 1-2 Layers of TCP/IP 5-Layer model together.

    5-Layer TCP/IP vs. 4-Layer TCP/IP vs. 7-Layer OSI Networking Model

    1. Physical Layer
    2. Data link Layer
    3. Network Layer

    1. Session Layer
    2. Presentation Layer
    3. Application Layer

    Communication protocols enable an entity in one host to interact with a corresponding entity at the same layer in another host.

    Service definitions, like the OSI Model, abstractly describe the functionality provided to an (N)-layer by an (N-1) layer, where N is one of the seven layers of protocols operating in the local host.

    At each level N, two entities at the communicating devices (layer N peers)
    exchange protocol data units (PDUs)
    by means of a layer N protocol.

    Each PDU contains a payload, called the service data unit (SDU),
    along with protocol-related headers or footers.

    Data processing by two communicating OSI-compatible devices proceeds as follows:

    1. The data to be transmitted is composed at the topmost layer of the transmitting device (layer N) into a protocol data unit (PDU).
    2. The PDU is passed to layer N-1, where it is known as the service data unit (SDU).
    3. At layer N-1 the SDU is concatenated with a header, a footer, or both, producing a layer N-1 PDU. It is then passed to layer N-2.
    4. The process continues until reaching the lowermost level, from which the data is transmitted to the receiving device.
    5. At the receiving device the data is passed from the lowest to the highest layer as a series of SDUs while being successively stripped from each layer’s header or footer, until reaching the topmost layer, where the last of the data is consumed.

    Data Link layer Packet Format

    1. Header, payload, trailer are the main parts of a frame.
    2. Header contains the number of characters in a frame & source and destnation address.
    3. Payload (or packet) contains the data to be sent.
    4. Trailer is the end of packet and it also contains the starting and ending bits.
    5. This frame is sent to network layer.

    Segment vs. Packet vs. Frame

    Irrespective of whether you are talking about network layer, transport layer or physical layer, the term “PDU (Protocol Data Unit)” is generic and applies to all.

    Of Course different layers add different fields on top of data, but they can still be called PDUs. Being said that, according to OSI reference layered architecture, below is what is widely accepted.

    1. PDU of Transport layer is called a Segment
      Port numbers are part of Segments
    2. PDU of network layer is called a Packet in OSI and Datagram in TCP/IP
      IP addresses are part of a Packet
    3. PDU of data link layer is called a Frame
      Mac addresses are part of Frames

    OSI Protocol Stack PDU Unit encapsulation

    TCP/IP Protocol Stack PDU Unit encapsulation

    Protocols

    TCP UDP
    Reliability Reliable
    TCP provides mechanisms to ensure that data is reliably delivered (UDP does not).
    UDP is not reliable, but reliability can added on application layer. An application can use UDP and can be reliable by using a timeout and resend at the application layer.
    Speed TCP is slow as it requires 3-way handshake. The load on DNS servers is also an important factor. DNS servers (since they use UDP) don’t have to keep connections. UDP is much faster.

    IPv4 addressing uses the concept of classes. In classful addressing, the address space is divided into five classes:A, B, C, D, and E. We can find the class of an address when the given address is in binary notation or dotted-decimal notation.

    IPv4 Classes

    For the IP address that most people are familiar with (IPv4), there are 4 sets (octets) of numbers, each with values of up to 255.

    Rules for IPv4 Classes:

    1. There must be no leading zero (e.g., 045).
    2. There can be no more than four numbers in an IPv4 address.
    3. Each number needs to be less than or equal to 255.
    4. A mixture of binary notation and dotted-decimal notation is not allowed.

    IP classes are primarily differentiated by the number of potential hosts they can support on a single network.

    The more networks supported on a given IP class, the fewer addresses are available for each network.

    Class A networks run up to 127.x.x.x (with the exception of 127.0.0.1, which is reserved for loopback or localhost connections).

    These networks are usually reserved for the very largest of customers, or some of the original members of the Internet and xkcd has an excellent map (albeit a bit dated) located here showing who officially owns what.

    Class B (128.x to 191.x) and Class C (192.x to 223.x) networks are much more fuzzy at the top level about who officially owns them.

    Class C addresses are primarily reserved for in-house networks which is why so many different manufacturers use 192.x as their default setting.

    Class D and E are reserved for special uses and normally are not required knowledge.

    IP belongs to the Network Layer (layer 3) in the OSI model.

    Internet protocol is working in network layer of OSI model in connection with tcp tx layer protocol.

    IP is a best-effort protocol, because it will make every effort to always transmit a datagram and also datagrams will not be just discarded.

    However, the delivery of the datagram to the destination is not guaranteed.

    1. IP is a unreliable protocol because it does not guarantee the delivery of a data-gram to its destination.
    2. The reliability must be provided by the upper layer protocols like TCP.
    3. IP does not support flow control, re-transmission, acknowledgement and error recovery.

    The next generation IP, or IPv6 has some advantages over IPv4:

    1. Larger Address Space: An IPv6 address is 128 bytes long where IPv4 has 32-bit address only, this is a huge increase in the address space.
    2. Better Header Format: IPv6 uses a new header format which simplifies and speeds up the routing process because most of the options do not need to be checked by routers.
    3. New Options: IPv6 has new options to allow for additional functionalities.
    4. Allowance for extension: In IPv6 new technologies can easily embedded.
    5. Support For Resource Allocation: In IPv6, flow label has been added to enable the source to request special handling of the packet. This mechanism can be used to support traffic such as real-time audio and video.
    6. Support For More Security: The encryption and authentication options in IPv6 provide confidentiality and integrity of the packet.

    SSH or Secure Shell is most well known by Linux users, but has a great deal that it can be used for. SSH is designed to create a secure tunnel between devices, whether that be systems, switches, thermostats, toasters, etc. SSH also has a unique ability to tunnel other programs through it, similar in concept to a VPN so even insecure programs or programs running across unsecure connections can be used in a secure state if configured correctly. SSH runs over TCP port 22.

    FTP or File Transfer Protocol, is one of the big legacy protocols that probably should be retired.

    1. In FTP protocol, client contacts server using TCP as the transport protocol.

      Client initiates both the control and data connections in passive mode.

    2. In file transfer protocol, data transfer can be done in
      1. stream mode
      2. block mode
      3. compressed mode
    3. FTP is built on client server architecture.
    4. FTP is primarily designed for large file transfers, with the capability of resuming downloads if they are interrupted.
    5. Anonimous FTP vs. Standard Login

      Access to an FTP server can be accomplished using two different techniques:

      1. Standard Login
        requires an active user to login
      2. Anonymous access
        User does not need to have an account or password.
        The login used is anonymous or guest, with the password usually requesting the user’s ID.
        It enables a large number of users to access the files on the host without needing to set up a login for all of them.
        It exercises a strict control over the areas an anonymous user can access.
    6. Big problem with FTP is that the credentials of the user are transmitted in cleartext which means that anybody listening on the wire could sniff the credentials extremely easily.

      Two competing implementations of FTP that take care this issue are SFTP (FTP over SSH) and FTPS (FTP with SSL).

    Dynamic Host Configuration Protocol is the default way for connecting up to a network. The implementation varies across Operating Systems, but the simple explanation is that there is a server on the network that hands out IP addresses when requested. Upon connecting to a network, a DHCP request will be sent out from a new member system. The DHCP server will respond and issue an address lease for a varying amount of time. If the system connects to another network, it will be issued a new address by that server but if it re-connects to the original network before the lease is up- it will be re-issued that same address that it had before. To illustrate this point, say you have your phone set to wifi at your home. It will pick up a DHCP address from your router, before you head to work and connect to your corporate network. It will be issued a new address by your DHCP server before you go to starbucks for your mid-morning coffee where you’ll get another address there, then at the local restaurant where you get lunch, then at the grocery store, and so on and so on.

    ICMP is the Internet Control Message Protocol. Most users will recognize the name through the use of tools such as ping and traceroute, as this is the protocol that these services run over among other things. Its primary purpose is to tell systems when they are trying to connect remotely if the other end is available. Like TCP and UDP, it is a part of the IP suite and uses IP port number 1. Please note, this is not TCP port 1 or UDP port 1 as this is a different numbering scheme that for reference can be located here (For your reference, TCP uses IP port 6, while UDP uses IP port 17). That being said, different functions of ICMP use specific ports on TCP and UDP. For example, the ‘echo’ portion of ping (the part where someone else is able to ping you) uses TCP port 7.

    SNMP is the “Simple Network Management Protocol”. Most systems and devices on a network are able to tell when they are having issues and present them to the user through either prompts or displays directly on the device. For administrators unfortunately, it can be difficult to tell when there is a problem unless the user calls them over. On devices that have SNMP enabled however, this information can be broadcast and picked up by programs that know what to look for. In this way, reports can be run based on the current status of the network, find out what patches are current not installed, if a printer is jammed, etc. In large networks this is a requirement, but in any size network it can serve as a resource to see how the network is fairing and give a baseline of what its current health is.

    1. SMTP is a push protocol; it pushes the message from the client to the server.
    2. For sending email SMTP is used two times, between the sender and the sender’s mail server and between the two mail servers.
    3. To send mail, a system must have the client MTA (Message transfer agents), and to receive mail, a system must have a server MTA.
    4. SMTP uses commands and responses to transfer messages between an MTA client and an MTA server. Commands are sent from the client to the server and Responses are sent from the server to the client.
    5. SMTP is not used in pulling the message.

    Post Office Protocol, version 3 is used for accessing the message.

    POP3 is very simple protocol and has limited functionality.
    A POP3 server listens on well-known port 110.

    POP3 supports two types of modes:

    1. Delete Mode: In this mode, the mail is deleted from the mailbox after each retrieval.
    2. Keep Mode: The mail remains in the mailbox after retrieval.

    IMAP4 (Internet Mail Access Protocol) is used in Internet for accessing the mail. IMAP4 has more features than POP3 and is more powerful.
    IMAP4 vs. POP3

    Asynchronous Transfer Mode (ATM) is the cell relay protocol.

    Design Goals of ATM:

    1. A technology is needed to take advantage of high rate transmission media and less susceptible to noise degradation for maximizing the data rates.
    2. The system must interface with existing systems and provide wide-area inter-connectivity between.
    3. The implementation of design must be cheaper enough so that everyone can afford it. If ATM is to become the backbone of international communications, as indeed, it must be available at low cost to every user who wants it.
    4. The new system must be able to work with and support the existing telecommunication hierarchies.
    5. The new system must be connection-oriented to ensure accurate and predictable delivery.
    6. One objective is to move as many of the functions to hardware as possible (for speed) an eliminate as many software functions as possible (again for speed).

    APIs

    Services are programs that run in the background based on a particular system status such as startup. Services exist across nearly all modern operating systems, although vary in their naming conventions depending on the OS- for example, services are referred to as daemons in Unix/Linux-type operating systems. Services also have the ability to set up actions to be done if the program stops or is closed down. In this way, they can be configured to remain running at all times.

    The ability to remote into servers without having to actually be there is one of the most convenient methods of troubleshooting or running normal functions on a server- Terminal Services allow this capability for admins, but also another key function for standard users: the ability to run standard applications without having to have them installed on their local computers. In this way, all user profiles and applications can be maintained from a single location without having to worry about patch management and hardware failure on multiple systems.

    Web Service API
    Definition Web Service is

    1. a collection of APIs working together to perform a particular task.
    2. Supports interoperable machine-to-machine interaction over a network
    3. Clients and servers most commonly communicate over the HTTP protocol.
    4. Uses a standardized messaging system.

    Examples: Google Maps API, Amazon Advertising API, Twitter API, YouTube API, etc.

    API
    1. Stands for Application Programming Interface.
    2. Allows an application to interact with certain system/application/library/etc.

    Examples: there are API’s for OSs (WinAPI), API’s for other applications (like databases) and for specific libraries (for example, image processing), etc.

    Communication endpoints

    1. Both are means of communications, a messenger, software intermediary
      that delivers your request to the destination that you’re requesting it from
      and then delivers the response back to you.
    2. All Web Services are API
      but not all APIs are Web Services.
    1. Web service facilitates interaction between two machines over a network
    2. In case of Web applications, the API use is web based.
    3. A Web service is merely an API wrapped in HTTP.
    1. An API acts as an interface between two different applications so that they can communicate with each other.
    2. Client-server relationship is not necessary: An API doesn’t always need to be web based, two pieces of software can be on the same or different machines.
    3. .
      E.g.: Desktop applications such as spreadsheets and word documents use VBA and COM-based APIs which don’t involve Web service.

      A server application such as Joomla may use a PHP-based API present within the server which doesn’t require Web service.

    Means/Style of communication Web service uses SOAP, REST, and XML-RPC as a means of communication.
    APIs may be exposed to in multiple ways.
    APIs may use any means of communication to initiate interaction between applications.
    E.g., the system calls are invoked using interrupts by the Linux kernel API.
    Functionality and Specifications
    1. Interface is depicted in a machine-processable format
      usually specified in Web Service Description Language (WSDL).
    2. A Web service might not contain a complete set of specifications and sometimes might not be able to perform all the tasks that may be possible from a complete API.
    1. An API is a set of protocols that describes how a data exchange should work.
    2. An API exactly defines the methods, has a complete set of rules and specifications for one software program to interact with the other.

    What is REST?
    1. REST stands for Representational State Transfer.
    2. REST is an architectural style for developing Web Services
    3. It is a set of constraints that ensure a scalable, fault-tolerant and easily adaptable system.
    4. It exploits the ubiquity of HTTP protocol and uses HTTP method to define actions.
    5. It relies on a stateless, client-server and cacheable communication protocol.
    6. It revolves around resource.
    7. In REST architecture, a REST Server provides access to resources and REST client accesses these resources.
    8. Each resource is identified by URIs or global IDs
    9. Every component being a resource can be accessed through a shared interface using standard HTTP methods.
    10. REST uses multiple ways to represent a resource, such as text, JSON, and XML.
      XML and JSON are currently the most popular representations of resources.
    11. REST is commonly used in social media, web chat, mobile services and Public APIs like Google Maps.

    1. REST is an architectural style for developing Web Services
    2. Web Services developed in the REST style are referred to as RESTful web services. These web services use HTTP methods to implement the concept of REST architecture. A RESTful web service usually defines a URI, Uniform Resource Identifier a service, provides resource representation like JSON and a set of HTTP methods.

    An API Contract IS the documentation of the API.

    It is a description of methods and argument as well as behavior and result objects.
    The documentation is the place where you declare how your API will behave.
    In this sense it the contract between you API provider and the developers that will use it around the world.

    API contract, is a common phrase to describe how API services are consumed.
    An API Contract is an agreement response format between an API and the client (mobile/web/application).
    An API provider and consumer can enter into a contract for services around a set of resources, in a self-service way.

    Often, the contract is specified as interface, if you develop a component.
    If you develop a service, the contract may be derived by accessing special methods of the service, like obtaining the WSDL to a Web Service.

    The contract is defined by the party which provides the service.

    The most common is OpenAPI Specification (formerly known as Swagger), RAML and BluePrint.
    OAS is slowly becoming the de facto standard and is the most common form of an API contract toda. OAS is an vendor neutral, portable, and open API description format, which standardizes how REST APIs are described.

    Contract testing is a way to ensure that services (such as an API provider and a client) can communicate with each other.

    1. REST architecture treats any content as a resource
    2. It can be either text files, HTML pages, images, videos or dynamic business information.
    3. REST Server gives access to resources and modifies them.
    4. Each resource is identified by URIs/ global IDs.

    REST uses different representations to define a resource like text, JSON, and XML.
    XML and JSON are the most popular representations of resources.

    RESTful web services use the HTTP protocol as a medium of communication between the client and the server.

    REST is stateless, therefore the SERVER has no status (or session data).

    With a well-applied REST API, the server could be restarted between two calls, since all data is transferred to the server.

    REST Web service uses POST method primarily to perform operations and GET for accessing resources.

    CRUD Operations in REST are represented by the following HTTP methods:
    Post – create
    Get and Head – read
    Put and Patch – update
    Delete – delete

    Options – used to describe the communication options for the target resource.

    Read more about HTTP Methods >>>

    URI
    1. URI stands for Uniform Resource Identifier
    2. According to RFC 3986, URI is a compact sequence of characters that identifies an abstract or physical resource.
    3. URI is the superset of both URL and URN.
      Both URLs and URNs fall under the umbrella of URI.
    4. RFC 3986 says a URI can identify a web resource (e.g. an image, video, CSS file, etc) by location, name, or both.
    5. “location” is the access mechanism used to retrieve a particular resource,
      e.g., http://, ftp://, etc.
    6. “name” refers to the globally unique name of the resource.
      E.g., given the following web address: https://foobar/path/to/image.jpg,
      the foobar/path/to/image.jpg section of the address refers to the name, or URN.
    7. URI vs. URL: relevant RFCs are contradictory.
      1. URIs technically begins with a scheme name, (RFC 3986, section 1.1.1)
        (which in many cases tells you exactly how to access the resource).

        • ftp://ftp.is.co.za/rfc/rfc1808.txt
        • http://www.ietf.org/rfc/rfc2396.txt
        • ldap://[2001:db8::7]/c=GB?objectClass?one
        • mailto:John.Doe@example.com
        • news:comp.infosystems.www.servers.unix
        • tel:+1-816-555-1212
        • telnet://192.0.2.16:80/
        • urn:oasis:names:specification:docbook:dtd:xml:4.1.2
      2. but the RFC also says that URIs can be a name, locator, or both (RFC 3986, section 1.1.3).
    URL

    Uniform Resource Locator, is a widely used term across the Internet to define a particular web address.

    1. (URL) refers to the subset of URIs that (RFC 3986, section 1.1.3) provide a means of locating the resource by describing its primary access mechanism (e.g., its network “location”).
      This is achieved by defining a protocol for the web resource.
    2. URI vs URN:
      A URL defines where the resource can be obtained,
      unlike a URN which simply defines the resource’s name.
    3. All URLs can be considered URIs, however not all URIs are considered URLs.
    4. URI vs. URL:
      The URI/URL debate ranks among the top three technical religious wars.

      No matter how the URL is structured, it will always be technically correct if you use the term URI.

      URLs always include a network protocol e.g. HTTP, HTTPS, FTP etc to retrieve a resource from its location.

      When the resource contains a specific location identifier (a protocol such as http://, ftp:// etc.) the term URL is often used as it is more specific.

    URN
    1. URN stands for Universal Resource Name
    2. URN is used to identify a resource by a unique name
    3. The downfall of a URN is that it does not provide a location to retrieve a particular web resource.
    4. A URN uses a namespace and a namespace identifier to identify the resource in question.
    5. According to W3 URI Clarification, a URN namespace comes in the form of urn:isbn:n-nn-nnnnnn-nwhere isbn would be the namespace identifier.

      ISBN numbers used to identify any book uniquely are a good example of URN.

    6. URNs can also be used to identify documents. In this case, the URN would need to be translated into a URL by a resolver where the document would be subsequently downloaded from a web server.
    7. Fragments like file.htm actually are not URNs, because URNs are required to use a special notation with urn: in the beginning.
    8. URN vs. URL: URN is completely different from URL as it doesn’t include any protocol.
    Examples
    1. mailto:john.doe@example.com (URL due to the mailto: protocol)
    2. http://www.ietf.org/rfc/rfc2396.txt (URL due to the http:// protocol)
    3. urn:ietf:rfc:2648 (URN as it does not specify a locator)
    4. urn:isbn:0508672084 (URN as it does not specify a locator)

    1. URI stands for Uniform Resource Identifier.
    2. It is a string of characters designed for unambiguous identification of resources and extensibility via the URI scheme.
    3. The purpose of a URI is to locate a resource(s) on the server that is hosting a web service.

    A URI’s format is:

    protocol://serviceName/ResourceType/ResourceID
    

    The “payload” is the data you are interested in transporting.

    This is differentiated from the things that wrap the data for transport like the HTTP Request/Response headers, authentication, etc.

    Read how encapsulation happens at each layer of the networking model >>>

    Generally, REST is preferred due to its simplicity, performance, scalability, and support for multiple data formats.

    However, SOAP is favorable to use where service requires an advanced level of security and transactional reliability.

    REST SOAP
    Representational State Transfer Simple Object Access Protocol
    Architectural Style

    Web Service can only be treated as RESTful if it follows the constraints of being:

    1. Client-server
    2. Stateless
    3. Cacheable
    4. Layered System
    5. Uniform Interface
    XML-based protocol
    SOAP was designed with a specification.
    It includes a WSDL file which includes the information on what the Web Service does in addition to the location of the Web Service
    REST can make use of SOAP as the underlying protocol for Web Services since it is just an architectural pattern SOAP cannot make use of REST since SOAP is a protocol and REST is an architectural pattern
    Does the service expose data or business logic? REST is commonly used for exposing data SOAP is used for exposing logic.
    formal contract If there is a requirement from clients or providers for a formal contract, SOAP can provide contract via WSDL.
    Platform and language independent
    REST uses HTTP headers to hold meta information.
    Since REST uses standard HTTP, it is much simpler.
    SOAP runs on HTTP but envelops the message
    data output types REST supports multiple data output types, including XML, CSV, and JSON.

    RESTful service returns various MediaTypes for the same resource, depending on the request header parameter “Accept” as application/xml or application/json for POST
    and /user/1234.json or GET /user/1234.xml for GET.

    SOAP can only handle XML.
    Support for AJAX calls REST can apply the XMLHttpRequest
    Synchronous and asynchronous calls REST has built-in support for synchronous SOAP enables both synchronous/ asynchronous operations
    Speed REST is generally faster and uses less bandwidth.

    Because the JSON format is easier to parse than XML, using REST to send data in JSON can actually save on computer infrastructure costs by requiring less computing power to do the same job.

    SOAP is slower than REST
    Scalability REST has better performance and scalability.
    ST in REST comes from State Transfer. You transfer the state around instead of having the server store it, this makes REST services scalable.
    Caching REST reads can be cached

    REST isable to cache data transfers, so when another endpoint requests an already completed query, the API can use the data from the previous request.
    REST provides superior performance, particularly through caching for information that’s not altered and not dynamic.

    SOAP reads cannot be cached

    SOAP implementations have to process the query every time.

    Security SSL SSL and WS-Security
    SOAP offers better support for Web Services specifications, often making it a stronger option when standardization and security are primary concerns.

    While both SOAP and REST support SSL (Secure Sockets Layer), SOAP it also supports WS-Security which adds some enterprise security features.

    If the security is a major concern and the resources are not limited then we should use SOAP web services. , then we should go with SOAP as here high security is needed.
    E.g., when creating a web service for payment gateways (dealing with crucial private information like bank account numbers), financial and telecommunication related work etc.

    However, SOAP’s extra security isn’t necessary if you’re sending the day’s forecast to a mobile application.

    WS-AtomicTransaction If you need ACID Transactions over a service, you’re going to need SOAP.
    Client REST services are meant to be called by the client-side application and not the end user directly.
    REST allows better support for browser clients due to its support for JSON.
    SOAP is like a custom desktop application, closely connected to the server
    Stateless or Stateful calls REST is naturally stateless.
    If we need Stateless CRUD operations then go with REST.
    SOAP has built-in stateful operations.
    SOAP is designed to support conversational state management.
    WS-ReliableMessaging REST doesn’t have a standard built-in messaging system.

    If a communication fails, the client has to deal with it by retrying.

    SOAP offers built-in retry logic to compensate for failed communications.
    If your application needs Asynchronous processing and a guaranteed level of reliability and security, REST is the answer.
    Ease of use REST is easier to use for the most part and is more flexible.
    REST is easier to implement, requires less bandwidth and resources.
    It’s easier to integrate with existing websites with no need to refactor site infrastructure.
    This enables developers to work faster rather than spend time rewriting a site from scratch.
    Instead, they can simply add additional functionality.
    SOAP is not very easy to implement and requires more bandwidth and resources

    1. SOAP is used to provide a user interface that can be accessed by the client object, and the request that it sends goes to the server, which can be accessed using the server object.
    2. The user interface creates some files or methods consisting of server object and the name of the interface to the server object.
    3. It also contains other information such as the name of the interface and methods.
    4. It uses HTTP to send the XML to the server using the POST method, which analyzes the method and sends the result to the client.
    5. The server creates more XML consisting of responses to the request of user interface using HTTP.
    6. The client can use any approach to send the XML, like the SMTP server or POP3 protocol to pass the messages or reply to queries.

    PutAddress(): It is used to enter an address in the webpage and has an address instance on the SOAP call.
    PutListing(): It is used to allow the insertion of a complete XML document into the web page. It receives the XML file as an argument and transports the XML file to XML parser liaison, which reads it and inserts it into the SOAP call as a parameter.
    GetAddress(): It is used to get a query name and gets the result that best matches a query. The name is sent to the SOAP call in the form of text character string.
    GetAllListing(): It is used to return the full list in an XML format.

    1. When using SOAP, users often see the firewall security mechanism as the biggest obstacle.
      It blocks all the ports leaving few like HTTP port 80 and the HTTP port used by SOAP that bypasses the firewall.
    2. The technical complaint against SOAP is that it mixes the specification for message transport with the specification for message structure.

    There are two different methods available for developing SOAP-based web services, which are explained below:

    Contract-first approach: the contract is first defined by XML and WSDL, and then Java classes are derived from the contract.
    Contract-last approach: Java classes are first defined, and then the contract is generated, which is normally the WSDL file from the Java class.
    “Contract-first” method is the most popular approach.

    It is a common XML document that contains the elements as a SOAP message:

    1. Envelope:
      It is an obligatory root element that translates the XML document and defines the beginning and end of the message.
    2. Header:
      It is an optional item which contains information about the message being sent.
    3. Body:
      It contains the XML data comprising the message being sent.
    4. Fault:
      It provides the information on errors that occurred while during message processing.

    1. Must use encoded XML
    2. Envelope namespace must be used
    3. Encoding namespace must be used
    4. Must not consist of a DTD reference
    5. Must not have XML processing instruction

    1. SOAP message has a root Envelope element
    2. Envelope is an obligatory part of the SOAP message.
    3. If an envelope includes a header element, it should not contain more than one.
    4. Envelope version will change if the SOAP version changes.
    5. The SOAP envelope is indicated by the prefix ENV and the envelope element.
    6. The optional SOAP encoding is also specified using a namespace and the optional encoding style element.

    1. Application layer and transport layers of a network are used by SOAP
    2. HTTP and SMTP are the valid protocol of the application layer used as the transport for SOAP.
    3. HTTP is more preferable, since it works well with the current Internet infrastructure, in particular with firewalls.

    The SOAP requests can be sent using an HTTP GET method while the specification only contains details about HTTP POST.

    The SOAP protocol is used to provide simple access methods for all the applications available on the Internet, providing the following functionalities:

    Call A class which provides the main functionality for a remote method for which a call is needed.
    It is used to create the call() and to specify the encoding style of the registry that will be assigned when if necessary.
    This call() function is used by the RPC call, which represents the options of the call object.
    Deployment Descriptor A class used to provide the information about the SOAP services.
    It enables easy deployment without the need for other approaches.
    DOM2 Writer A class that serializes and uses DOM node as XML string to provide more functionalities.
    RPC Message A class used as the base class that calls and replies to the request submitted to the server.
    Service Manager A class that provides, lists and then outputs all SOAP services.

    HTTPUtils Provide the functionality of the POST method to safely meet the requirements.
    Parameter It is an argument for an RPC call used by both the client and the server.
    Response It is an object that represents an RPC reply from both client and server, but the result will not be displayed until after the method call.
    TCPTunnel It is an object that provides the ability to listen on a specific port and to forward all the host and port names.
    TypeConverter It helps to convert an object of one type into another type and this is called using the class in the form object.

    1. The security model includes the given security tokens. These tokens comprise digital signatures for protection and authentication of SOAP messages.
    2. Security tokens can be used to provide the bond between authentication secrets or keys and security identities.
    3. Security token uses the authentication protocols and an X.509 certificate to define the relationship between the public key and identity key.
    4. The signatures are used to verify the messages and their origin, generate knowledge to confirm the security tokens to bind the identity of a person to the identity of the originator.
    5. Security model prevents different attacks and can be used to protect the SOAP architecture.

    1. Top down SOAP Web Services
      1. Include creating WSDL document to create a contract between the web service and the client, with a required code as an option.
      2. This is also known as Contract-first approach.
      3. The top-down approach is difficult to implement because classes must be written to confirm the contract defined in WSDL.
      4. One of the benefits of this method is that both client and server code can be written in parallel.
    2. Bottom up SOAP Web Services
      1. Require the code to be written first and then WSDL is generated.
      2. It is also known as Contract-last approach.
      3. Since WSDL is created based on the code, bottom-up approach is easy to implement and client codes must wait for WSDL from the server side to start working.

    1. SOAP is both platform and language independent.
    2. SOAP separates the encoding and communications protocol from the runtime environment.
    3. Web service can retrieve or receive a SOAP user data from a remote service, and the source’s platform information is completely independent of each other.
    4. Everything can generate XML, from Perl scripts through C++ code to J2EE app servers.
    5. It uses XML to send and receive messages.
    6. It uses standard internet HTTP protocol.
      SOAP runs over HTTP; it eliminates firewall problems. When protocol HTTP is used as the protocol binding, an RPC call will be automatically assigned to an HTTP request, and the RPC response will be assigned to an HTTP reply.
    7. Compared to RMI, CORBA and DCOM, SOAP is very easy to use.
    8. SOAP acts as a protocol to move information in a distributed and decentralized environment.
    9. SOAP is independent of the transport protocol and can be used to coordinate different protocols.

    1. Speed
      SOAP is typically significantly slower than other types of middleware standards, including CORBA, because SOAP uses a detailed XML format. A complete understanding of the performance limitations before building applications around SOAP is hence required.
    2. SOAP is usually limited to pooling and not to event notifications when HTTP is used for the transport. In addition, only one client can use the services of one server in typical situations.
    3. If HTTP is used as the transport protocol, firewall latency usually occurs since the firewall analyzes the HTTP transport. This is because HTTP is also leveraged for Web browsing, and so many firewalls do not understand the difference between using HTTP within a web browser and using HTTP within SOAP.
    4. SOAP has different support levels, depending on the supported programming language. For instance, SOAP supported in Python and PHP is not as powerful as it is in Java and .NET

    WebSockets

    WebSockets are really just an extension of the socket idea.

    It was created due to HTTP limitations:

    1. How HTTP handled connections:
      Whenever you made a request, say to download html, or an image, a port/socket was opened, data was transferred, and then it was closed.

      The opening and closing creates overhead, and for certain applications, especially those that want rapid responses or real time interactions or display streams of data, this just doesn’t work.

    2. HTTP “pull” paradigm:
      The browser would request or pull information from servers, but the server couldn’t push data to the browser when it wanted to. This means that browsers would have to poll the server for new information by repeating requests every so many seconds or minutes to see if there was anything new.

      Unlike HTTP, the socket that is connected to the server stays “open” for communication. That means data can be “pushed” to the browser in realtime on demand.

    Read more about WebSockets here >>>

    REST

    Creating REST endpoints using HTTP is how the vast majority of Web applications (whether it’s Ruby, Java, Go, NodesJS etc.) are architected.

    Read more about REST here >>>

    REST is by far the most standardized way of structuring the API for Requests. But since it involves using HTTP is also has the overhead associated with that protocol. For most applications, information only needs to be transferred when a user takes an action. For instance, when browsing a news site, once the browser has requested the article, the user is busy reading it, and not taking actions. Having the port-socket close during this time is actually saving resources. With less frequent interaction, HTTP works very well, and it is why it is used.

    For more real time interaction, or real time transfer or streaming of data, HTTP and REST aren’t the best suited protocol and abstraction combination. This is where Sockets and WebSockets shine.

    Microservices are a software development technique — a variant of the service-oriented architecture (SOA) architectural style that structures an application as a collection of loosely coupled services.

    Even though there is no official definition of what microservices are, a consensus view has evolved over time, in the industry.

    Services are small in size, messaging-enabled, bounded by contexts, autonomously developed, independently deployable, decentralized and built and released with automated processes.

    1. Services in a microservice architecture (MSA) are often processes that communicate over a network to fulfill a goal using technology-agnostic protocols such as HTTP.
    2. business-driven development (e.g. domain-driven design):
      Services are organized around fine-grained business capabilities.
      The granularity of the microservice is important – because this is key to how this approach is different from SOA.
    3. Microservice architecture adheres to principles such as fine-grained interfaces (to independently deployable services)
    4. The benefit of decomposing an application into different smaller services is that it improves modularity.
    5. This makes the application easier to understand, develop, test, and become more resilient to architecture erosion.
    6. It parallelizes development by enabling small autonomous teams to develop, deploy and scale their respective services independently.
    7. It also allows the architecture of an individual service to emerge through continuous refactoring.
    8. Microservice-based architectures enable continuous delivery and deployment. A change to a small part of the application only requires rebuilding and redeploying only one or a small number of services.
    9. Services can be implemented using different programming languages, databases, hardware and software environment, depending on what fits best.

      This does not mean that a single microservice is written in a patchwork of programming languages. While it is almost certainly the case that different components that the service is composed of will require different languages or APIs (example, the web server layer may be in Java or Javascript, but the database may use SQL to communicate to an RDBMS), this is really reflective of a comparison to the monolithic architecture style.

      If a monolithic application were to be re-implemented as a set of microservices, then the individual services could pick their own implementation languages.

      So one microservice could pick Java for the web layer, and another microservice could pick a Node.js-based implementation, but within each microservice component, the implementation language would be uniform.

    10. A microservice is not a layer within a monolithic application (example, the web controller, or the backend-for-frontend).
      Rather it is a self-contained piece of business functionality with clear interfaces, and may, through its own internal components, implement a layered architecture.

      From a strategy perspective, microservices architecture essentially follows the Unix philosophy of “Do one thing and do it well”.

    11. It is quite common for such an architectural style to be adopted for cloud-native applications, and applications using lightweight container deployment.
    12. Because of the large number (when compared to monolithic application implementations) of services, decentralized continuous delivery and DevOps with holistic service monitoring are necessary to effectively develop, maintain, and operate such applications.
    13. A consequence of (and rationale for) following this approach is that the individual microservices can be individually scaled.

      In the monolithic approach, an application supporting three functions would have to be scaled in its entirety even if only one of these functions had a resource constraint.

      With microservices, only the microservice supporting the function with resource constraints needs to be scaled out, thus providing resource and cost optimization benefits.

    Read more on Wikipedia >>>

    JSON XML
    JavaScript Object Notation extensible markup language
    verbosity Less verbose XML uses more words
    model’s structure JSON data model’s structure matches the data: JSON’s data structure is a map (just key/value pairs) XML is a tree
    Both formats are meant to provide a simple protocol for sending data, and neither use encryption by default
    Security JSON is mentioned to be less secure in comparison to XML.
    These insecurities have to do with the way JSON can (but should never be) parsed in Javascript.
    JavaScript treats JSON strings as it’s cup of tea and tries to interpret it directly, which give workarounds to attackers to fool JavaScript interpreter to run malicious JavaScript embedded in the string causing vulnerabilities, while XML has to go through a parsing stage to be parsed into an object making it harder to attack with. 2 such JavaScript functionalities are eval() method and tag.
    JSON is based on the syntax for coding objects in javascript, so evaluating a JSON result in javascript returns a valid object.
    This may open JSON to a variety of javascript injection exploits.
    The way to resolve this: don’t use eval() to parse JSON in javascript, use a JSON parser and fix any security issues in your server that allow unescaped user generated content in the response.
    XML parsers in their default configuration are open to XXE injection attacks and DTD validation attacks, so XML data exchanges need to be hardened if used.
    Speed JSON is generally considered faster, since it takes less characters to assemble.
    Ease of use JSON is easier to use
    JASON is easy to use in JavaScript, since JSON is simply JavaScript Object Notation, and all JavaScript Objects can be converted to or from JSON representation.
    Readability Many (especially newer) developers prefer using XML because of its readability. It is structured in such a way that it is much easier for a human to read through it. This of course is what makes it bulkier than JSON.

    Marshalling – process of converting the data or the objects into a byte-stream.
    Unmarshalling is the reverse process of converting the byte-stream back to their original data or object.

    The conversion is achieved through “serialization”.

    Converting Objects to and from JSON Documents:

    1. Obj->JSON = Marshelling
    2. JSON->Obj = Unmarshelling

    1. Many APIs have a certain limit set up by the provider.
    2. It is important to estimate your usage and understand how that will impact the overall cost of the offering.
    3. Whether this will be a problem depends in large part on how data is leveraged.
    4. Getting caught by a quota and effectively cut-off because of budget limitations will render the service (and any system or process depending on it) virtually useless.

    1. APIs, especially modern RESTful APIs, are a nice creation that can certainly simplify and accelerate integration efforts, which makes it more likely you will benefit from them.
    2. But APIs can and do change for various reasons, sometimes abruptly, and hence REST APIs do not differ from traditional integration methods in this respect.
    3. If an API call is obsolete and disappears, your procedure will interrupt and it is important to understand how often the APIs you depend on change or are deprecated.

    Four common Web API architectural styles:

    1. HTTP for client-server communication
    2. XML/JSON as formatting language
    3. Simple URI as the address for the services
    4. Stateless communication

    1. Web API can be consumed by any clients which support HTTP verbs such as GET, PUT, DELETE, POST.
    2. Since Web API services do not require configuration, they can be easily used by any client.
    3. In fact, even portable devices such as mobile devices can easily use Web API, which is undoubtedly the biggest advantage of this technology.

    API Testing

    API testing is a type of software testing that involves testing application programming interfaces (APIs) directly and as part of integration testing to determine if they meet expectations for functionality, reliability, performance, and security.

    Since APIs lack a GUI, API testing is performed at the message layer and can validate application logic very quickly and effectively.

    API testing is critical for automation testing and CI/CD process because it can cope with short release cycles and frequent changes especially the presentation layer without breaking the test outputs.

    API testing also requires less maintenance effort compare to UI automation testing which makes it a preferred choice for Agile and DevOps teams.

    For Web and mobile applications, API often means Web services, and API testing refers to the automation test performed to the Web services.

    Commonly, applications have three separate layers or tiers including

    1. Presentation Layer or user interface
    2. Business Layer or application user interface for business logic processing
    3. and Database Layer for modeling and manipulating data.

    API Testing is performed at the most critical layer, the Business Layer, where business logic processing is carried out, and all transactions between User Interface and Database happen.

    API Testing - Business Layer

    Read more on Katalon Studio Web Site >>>

    Test pyramid (coined by Mike Cohn in his book Succeeding with Agile) shows three layers representing different types of testing.

    It suggests how much testing we should focus on at each layer. API and services tests are in the second layer as an important testing activity that we should focus on.

    Test Pyramid by Mike Cohn - Layers with Types of Testing

    1. Unit/component tests: This lowest level of testing brings the highest value and ROI. It is mainly performed by developers. The unit/component tests can attain between 70% and 80% of code coverage and require as much effort.
    2. Business rules / Functional tests: This level of testing focuses on business rules of the application under test. The tests are designed to test against user stories to ensure that all implemented functions are working as expected. In Web and mobile applications, data shown in the user interface is often returned from servers via API/services. So API testing is performed to ensure the accuracy of API/services. Testing at this level may need about 20% of the total testing effort.
    3. Workflow Tests (through the UI): functional UI testing is performed via the UI of the application to ensure that its features are built as expected. Due to the fact that automated functional UI testing is brittle, and any changes in the UI can break the tests, test automation teams should focus less on this level of testing.
    4. API testing yields the highest ROI compared to all other testing type performed by testers (Unit tests are performed mostly by developers).

    API testing is an important activity that testing teams should focus on.
    It offers a number of advantages over other kinds of testing:

    1. Test for Core Functionality (GUI independent)
      API testing provides access to the application without a user interface.
      The core and code-level of functionalities of the application will be tested and evaluated early before the GUI tests. Early testing will get feedback sooner and improve the team’s productivity.
      This will help to evaluate a build’s strengths and detect the minor issues which can become bigger during the GUI testing.
    2. Time Effective = Faster releases
      API testing usually is less time consuming than functional GUI testing.
      The web elements in GUI testing must be polled, which makes the testing process slower.

      API test automation requires less code so it can provide better and faster test coverage compared to GUI test automation. These will result in the cost saving for the testing project.

      It is common that executing a UI regression test suite takes 8-10 hrs while the same scenario with API testing takes only 1-2 hours. The flakiness of API testing is also lower than that of UI testing. All these allow us to release faster with API testing.

    3. Language independent
      In API testing, data is exchanged using XML or JSON. These transfer modes are completely language-independent, so any language can be used for automation, independent from the languages used to develop the application. XML and JSON are typically structured data so the verification is fast and stable. There are also built-in libraries to support comparing data using these data formats.
    4. Easy Integration with GUI
      API tests enable highly integrable tests, which is particularly useful if you want to perform functional GUI tests after API testing. For instance, simple integration would allow new user accounts to be created within the application before a GUI test started.
    5. Improved test coverage
      Most API/services have specifications, allowing us to create automated tests with high coverage, including functional testing (happy cases, negative cases) and non-functional testing. With virtual user techniques, we rarely find API test cases that could not be automated.

    Many protocols are now available to be used in API testing, such as JMS, REST, HTTP, UDDI and SOAP.

    1. The test environment of API is a bit complicated and requires the configuration of the database and server, depending on the software requirements.
    2. No GUI (Graphical User Interface) is available in this test form.
    3. When the installation process is complete, API is verified for the proper operation.
    4. Throughout the process, the API called from the original environment is set up with different parameters to study the test results.

    The five most important principles of an API test design are:

    1. Setup:
      Create objects, start services, initialize data, etc
    2. Execution:
      Steps to apply API or the scenario, including logging
    3. Verification:
      Oracles to evaluate the result of the execution
    4. Reporting:
      Pass, failed or blocked
    5. Clean up:
      Pre-test state

  • Choose the suite to add the API test case
  • Choose the test development mode
  • Demand the development of test cases for the required API methods
  • Configure the control parameters of the application and then test conditions
  • Configure method validation
  • Execute the API test
  • Check test reports and filter API test cases
  • Arrange all API test cases
  • During the API testing process, a request is raised to the API with the known data. This way you can analyze the validation response.

    While testing an API, you should consider:

    1. Accuracy of data
    2. Schema validation
    3. HTTP status codes
    4. Data type, validations, order and completeness
    5. Authorization checks
    6. Implementation of response timeout
    7. Error codes in case API returns
    8. Non-functional testing like performance and security testing

    The following factors should be considered when performing API testing:

    1. Defining the correct input parameters
    2. Verifying the calls of the mixture of two or more added value parameters
    3. Defining the basic functionality and scope of the API program
    4. Writing appropriate API test cases and making use of testing techniques such as equivalence class, boundary value, etc. to check the operability
    5. Testing case execution
    6. Comparing the test result with the expected result
    7. Verifying the API behavior under conditions such as connection to files and so on.

    There are myriad of different API testing tools available.

    A few of common tools are Katalon Studio, Postman, SoapUi Pro, Tricentis Tosca, Apigee, etc.

    Both Unit and API testing target source code.
    This means that if an API method uses code based in .NET, then another supporting tool must have .NET.

  • Unit testing and Functional testing
  • Load testing to test the performance under load
  • Discovery testing to list, create and delete the number of calls documented in API
  • Usability and Reliability testing to get consistent results
  • Security and Penetration testing to validate all types of authentication
  • Automation testing to create and run scripts that require regular API calls
  • End to end Integration and Web UI testing
  • API documentation testing to determine its efficiency and effectiveness
  • While there are certainly specialty tests, and no list will be comprehensive in this realm, most tests fit broadly into these following nine categories:

    Functional Testing Functional testing includes the testing of particular functions in the codebase. These features are the representation of specific scenarios to make sure the API functions are handled well within the planned parameters.
    UI testing UI testing is defined as a test of the user interface for the API and other integral parts. It is mainly about the UI functions, in which the interface is naturally graphical or rely on the command-line endpoint calls.

    Generally, the test is more likely of the interface which ties into the API rather than the API testing itself. Although UI testing is not a specific test of API in term of the codebase, this still provides overview information about the health, usability, and efficiency of both front-end and back-end.

    Load testing Load testing is generally done after a specific unit, or the whole codebase has been completed to check if the theoretical solutions work as the practical plan.

    Load testing is performed to ensure the performance under both normal and at peak conditions. To do so, load testing takes on some various scenarios:

    Baseline – tests the API against theoretical regular traffic that the API expects in ordinary usage;

    Theoretical maximum traffic – this is to make sure during the full load periods, the solutions respond to those requests properly;

    Overload test – to test the maximum capability according to the theory, then add 10-20% more on the peak traffic.

    Security testing Security, Penetration and Fuzz testing are the three separate elements of the security auditing process. They are used to ensure the API implementation is secure from external threats.

    Besides enclosing with penetration and fuzz testing, security testing also includes additional steps like the validation of encryption methodologies, and of the design of the access control for the API. It also contains the user rights management and validating authorization checks to access the resources.

    Penetration testing Penetration testing is considered the second test in the process if auditing. In this testing type, the users with limited API knowledge will try to attack to assess the threat vector from an outside perspective, which is about functions, resources, processes, or aim to the entire API and its components.
    Runtime/ Error Detection This testing type is related to the actual running of the API, particularly with the universal results of utilizing the API codebase. Generally, they focus on one of the below aspects:

    Monitoring: The runtime of the compiled code is tested for different implementation errors, handler failures, and other intrinsic issues to help ensure the codebase does not contain any insecurity.

    Execution Errors: The code needs to show the response to valid requests and marks the failure for invalid requests predictably and in a known way, just the same with valid requests.

    Resource Leaks: The requests which are invalid or commonly illegal will be submitted to the API to test the memory, resource, data, or operation leaks/insecurities

    Error Detection: The known failure scenarios will be used to test the code to make sure the errors are appropriately figured and resolved

    Fuzz testing Fuzz testing is another step in the whole security audit process. In this testing type, a vast amount of random data (referred to as “noise” or “fuzz” ), will be input into the system with the aim to try a forced crash or any negative behavior. This fuzz test will help test the API in term of the limits to prepare for the “worst case scenarios.”
    Interoperability and WS Compliance testing
    Validation Testing Validation testing is among the final steps and plays an essential role in the development process. Validation test helps verify the aspects of product, behavior, and efficiency.

    In other words, the validation testing can be seen as an assurance of the correct development against the stated user needs and requirements.

    API Testing Unit Testing
    Conducted by QA Team Conducted by Developers
    Mostly Black Box Testing White Box Testing
    Aimed to assess the full functionality of the system for it will be employed by the end-user (external developers who will use your API) Used to verify whether each unit in isolation performs as expected
    Often run after the build is ready and authors do not have the access to the source code Each of the code modules must be ensured to pass the unit test before being built by the developers

    API Testing UI Testing
    API enables communication between two separate software systems.
    A software system implementing an API contains functions or subroutines that can be executed by another software system.
    UI (User Interface) testing refers to testing graphical interface such as how users interact with the applications, testing application elements like fonts, images, layouts etc.
    UI testing basically focuses on look and feel of an application.

    1. Parameter Selection
    2. Parameter Combination
    3. Call sequencing
    4. Output verification and validation
    5. Another important challenge is providing input values, which is very difficult as GUI is not available in this case.

    API testing is now preferred over GUI testing and is considered as most suitable because:

    1. It verifies all the functional paths of the system under test very effectively.
    2. It provides the most stable interface.
    3. It is easier to maintain and provides fast feedback.

    1. Missing module errors
    2. Documentation errors
    3. Parameter validation errors
    4. And some standard error expectations as if the result is not so predicted then the occurrence of errors can be seen and for the same warnings are specified in the form of a message. There can be one or more warnings within an individual module.

    1. Missing or duplicate functionality
    2. Fails to handle error conditions gracefully
    3. Stress
    4. Reliability
    5. Security
    6. Unused flags
    7. Not implemented errors
    8. Inconsistent error handling
    9. Performance
    10. Multi-threading issues
    11. Improper errors

    API Documentation

    The API documentation is a complete, accurate technical writing giving instructions on how to effectively use and integrate with an API.

    It is a compact reference manual that has all the information needed to work with the API, and helps you answer all the API testing questions with details on functions, classes, return types, arguments, and also examples and tutorials.

    There are several available API documentation templates help to make the entire process simple and straightforward:

    1. Swagger
    2. Miredot
    3. Slate
    4. FlatDoc
    5. API blueprint
    6. RestDoc
    7. Web service API specification

    1. Source of the content
    2. Document plan or sketch
    3. Delivery layout
    4. Information needed for every function in the document
    5. Automatic document creation programs

    HTTP

    What is HTTP?
    1. HTTP stands for Hypertext Transfer Protocol.
    2. HTTP is a networking protocol for distributed, collaborative, hypermedia information systems.
    3. HTTP is the foundation of data communication for the World Wide Web.
    4. It is a set of rules for transferring of data on WWW (World Wide Web).
    5. The standards development of HTTP has been coordinated by
      1. the Internet Engineering Task Force (IETF)
      2. and the World Wide Web Consortium

      culminating in the publication of a series of Requests for Comments (RFCs),
      most notably RFC 2616 (June 1999),
      which defines HTTP/1.1, the version of HTTP in common use.

    The basic features of HTTP
    1. HTTP is a request and response protocol.
    2. HTTP is a media independent protocol.
    3. HTTP is a stateless protocol.

    1. In HTTP, there are two different roles: server and client.
    2. In general, the client always initiates the conversation; the server replies.
    3. HTTP is text based; that is, messages are essentially bits of text, although the message body can also contain other media. Text usage makes it easy to monitor an HTTP exchange.

    1. The HTTP client and HTTP server exchange information about resources identified by URLs.
    2. We say that the request and response contain a representation of the resource.
    3. By representation, we mean information, in a certain format, about the state of the resource or how that state should be in the future.
    4. Both the header and the body are pieces of the representation.
    5. The HTTP headers, which contain metadata, are tightly defined by the HTTP spec; they can only contain plain text, and must be formatted in a certain manner.
    6. The body can contain data in any format, and this is where the power of HTTP truly shines.
    7. You can send plain text, pictures, HTML, and XML in any human language.
    8. Through request metadata or different URLs, you can choose between different representations for the same resource.
    9. E.g., you might send a webpage to browsers and JSON to applications.

    Third Step.

    1. In first step client sends a segment to establish a connection with the server.
    2. In the second the step the client waits for the acknowledgement to be received from the server.
    3. After receiving the acknowledgement, the client sends actual data in the third step.

    1. HTTP messages are made of a header and a body.
    2. The body can often remain empty; it contains data that you want to transmit over the network, in order to use it according to the instructions in the header.
    3. The header contains metadata, such as encoding information; but, in the case of a request, it also contains the important HTTP methods.
    4. In the REST style, you will find that header data is often more significant than the body.
    HTTP request HTTP response
    1. Request line that consists of:

    • An action showing HTTP methods like GET, PUT, POST, DELETE.
    • Uniform Resource Identifier (URI), which is the identifier for the resource on the server.
    • HTTP Version

    E.g., GET /images/logo.png HTTP/1.1,
    which requests a resource called /images/logo.png from server

    2. Request Header,
    which carries metadata (as key-value pairs) for the HTTP Request Message.

    E.g., Accept-Language: en

    Metadata could be:

    • a client (or browser) type
    • format supported by the client
    • format of a message body format
    • cache settings, and so on
    3. Empty line

    The empty line must consist of only <CR><LF> and no other white-space.

    4. Optional Request Message Body

    Rules

    1. The request line and headers must all end with <CR><LF> (carriage return followed by a line feed).
    2. A request line containing only the path name is accepted by servers to maintain compatibility with HTTP clients before the HTTP/1.0 specification in RFC1945.
    3. In the HTTP/1.1 protocol, all headers except Host are optional.

    HTTP header fields allow the client and server to pass information with the request and response messages.

    Field Description
    General header It applies for both request and response message.
    Request header It contains information for the request message.
    Response header It is used to contain response header information sent by the web server.
    Entity header It is used to contain more information about the body of the entity.

    What is a “MIME type”, what does it consist of, and what is it used for?

    Electronic mail has a simple structure. It can send messages only in NVT 7-bit ASCII format.
    It cannot be used for those languages that are not supported by 7-bit ASCII characters so languages such as French, German, Hebrew, Russian, Chinese etc cannot use this format.
    Also, it cannot be used to send binary files or video or audio data.

    We need a protocol that allows non-ASCII data to be sent through e-mail.
    So Multipurpose Internet Mail Extensions (MIME) is the answer.

    1. MIME is an acronym for Multi-purpose Internet Mail Extensions.
    2. MIME transforms non-ASCII data at the sender site to NVT (Network Virtual Terminal) ASCII data and delivers them to the client MTA to be sent through the Internet. The message at the receiving side is transformed back to the original data. Main task of MIME is to transform non-ASCII data to ASCII data and vice versa.
    3. MIME is used as a standard way of classifying file types over the Internet.
    4. Web servers and browsers have a defined list of MIME types,
      which facilitates transfer of files of a known type,
      irrespective of operating system or browser.
    5. A MIME type actually has two parts: a type and a subtype that are separated by a slash (/).

      E.g., the MIME type for Microsoft Word files is application/msword (type is application and the subtype is msword).

    MIME-Version: 1.0
    Content-Type: multipart/mixed; boundary=frontier
    
    This is a message with multiple parts in MIME format.
    --frontier
    Content-Type: text/plain
    
    This is the body of the message.
    --frontier
    Content-Type: application/octet-stream
    Content-Transfer-Encoding: base64
    
    PGh0bWw+CiAgPGhlYWQ+CiAgPC9oZWFkPgogIDxib2R5PgogICAgPHA+VGhpcyBpcyB0aGUg
    Ym9keSBvZiB0aGUgbWVzc2FnZS48L3A+CiAgPC9ib2R5Pgo8L2h0bWw+Cg==
    --frontier--
    
    1. Each MIME message starts with a message header.
    2. The header contains information about the message content and boundary.
    3. MIME defines five types of headers:
      1. MIME-version
      2. Content-Type
      3. Content-Transfer-Encoding
      4. Content-Id
      5. Content-Description
    4. Content-Type: multipart/mixed; boundary=frontier

      means that message contains multiple parts
      where each part is of different content type
      and they are separated by –frontier as their boundary.

    5. Each part consists of its own content header (zero or more Content- header fields) and a body.
    6. Multipart content can be nested.
    7. The content-transfer-encoding of a multipart type must always be 7bit, 8bit, or binary to avoid the complications that would be posed by multiple levels of decoding.
    8. The multipart block as a whole does not have a charset; non-ASCII characters in the part headers are handled by the Encoded-Word system, and the part bodies can have charsets specified if appropriate for their content-type.

    1. Cross-origin resource sharing (CORS) is a mechanism that allows many resources (e.g., fonts, JavaScript, etc.) on a web page to be requested from another domain outside the domain from which the resource originated.
    2. It’s a mechanism supported in HTML5 that manages XMLHttpRequest access to a domain different.
    3. CORS adds new HTTP headers that provide access to permitted origin domains.
    4. For HTTP methods other than GET (or POST with certain MIME types),
      the specification mandates that browsers first use an HTTP OPTIONS request header
      to solicit a list of supported (and available) methods from the server.
    5. The actual request can then be submitted.
    6. Servers can also notify clients whether “credentials” (including Cookies and HTTP Authentication data) should be sent with requests.

    As per Wikipedia, Head-of-line blocking (HOL blocking) in computer networking is a performance-limiting phenomenon that occurs when a line of packets is held up by the first packet.

    Examples include input buffered network switches, out-of-order delivery (occurs when sequenced packets arrive out of order) and multiple requests in HTTP pipelining (number of allowed parallel requests in the browser is used up, and subsequent requests need to wait for the former ones to complete).

    Pipelining was introduced in HTTP/1.1 and was superseded by multiplexing via HTTP/2.

    As per Wikipedia, HTTP pipelining is a technique in which multiple HTTP requests are sent on a single TCP (transmission control protocol) connection without waiting for the corresponding responses.

    HTTP Pipelining

    1. Non-idempotent requests, like those using POST, should not be pipelined.
    2. Sequences of GET and HEAD requests can always be pipelined.
    3. A sequence of other idempotent requests like PUT and DELETE can be pipelined or not depending on whether requests in the sequence depend on the effect of others.

    As of 2018, HTTP pipelining is not enabled by default in modern browsers, due to several issues including buggy proxy servers and HOL blocking.
    Most HTTP proxies do not pipeline outgoing requests.

    The pipelining of requests results in a dramatic improvement in the loading times of HTML pages, especially over high latency connections such as satellite Internet connections.

    The speedup is less apparent on broadband connections, as the limitation of HTTP 1.1 still applies:
    the server must send its responses in the same order that the requests were received
    so the entire connection remains first-in-first-out (FIFO) and HOL blocking can occur.

    The asynchronous operation of HTTP/2 and SPDY are solutions for this.
    Browsers ultimately did not enable pipelining by default, and by 2017 most browsers supported HTTP/2 by default which used multiplexing instead.
    Multiplexing eliminates HOL blocking at the application layer, but HOL still exists at the transport (TCP)layer.

    Various types of Multiplexing
    1. Frequency Division Multiplexing (FDM)
    2. Time Division Multiplexing (TDM)
      • Synchronous TDM
      • Asynchronous TDM
    3. Wave Division Multiplexing (WDM)
    HTTP Multiplexing with SPDY

    HTTP Multiplexing is a method in HTTP/2 by which multiple HTTP requests can be sent and responses can be received asynchronously via a single TCP connection. Multiplexing is the heart of HTTP/2 protocol.

    Multiple HTTP Requests
    A single TCP connection can be used to make HTTP requests to a single origin only. For multiple origins multiple TCP connections are required.

    SPDY is the core part of HTTP/2 protocol, many of the http/2 protocol techniques are part SPDY.
    SPDY(speedy) is a network protocol which manipulates http protocol by compressing headers, predicting clients requests and other techniques, to fasten web experience.
    SPDY was developed by Google.

    HTTP/2 makes use of SPDY techniques, SPDY is deprecated.

    SPDY (pronounced “speedy”) is a open-specification networking protocol for transporting web content with particular goals of reducing web page load latency and improving web security.

    Google developed a SPDY-enabled Google Chrome browser and open-source web server. In lab tests, they compared the performance of these applications over HTTP and SPDY, and have observed up to 64% reductions in page load times in SPDY.
    Read more >>>

    SPDY supports all major browsers(firefox, chrome, opera, safari, ie).

    Why speedy is fast:

    1. Compression.
      Speedy compresses the headers, it will track whether the headers in this particular session has been sent and if sending headers was has already done then it is of no use to resend untilsession expires.
    2. Prefetching:
      If a client requests for an html file which includes some links(css stylesheets) then in http/1 protocols, the client should have to send request for getting those links(css stylesheets). In this case a speedy server could parse the html, include those links and send without waiting for client to ask for links.
    3. Priority:
      HTTP/1 handles limited parallel connections, like 6–8 based on browser, if connections are more, then we have to wait until previous connections are resolved, so even if there are connections that are important we have to wait. Speedy workarounds by using prioritized streams, so most urgent requests will be resolved first.
    4. Most importantly multiplexing:
      As said above about limited parallell connections, multiplexing can solve this, multiplexing basically means combining many signals into one, so the speedy browser combines its many requests into a single request and sends to server, the server then divides the requests(demultiplexing) into original number of requsts and responds.

    Speedy is fast because extra work is done by both client and server for processing the above tasks.
    Although http/2 has speedy at its core, http/2 can handle multi-host multiplexing, has faster encryption, has faster compression rate.

    Multiplexing vs. Pipelining

    HTTP Pipelining vs. HTTP Multiplexing
    HTTP requires that you deliver the entire response before any other request can proceed.
    With SPDY we can finally break the “head of line” requirement and deliver the responses as they become available.
    However, we also don’t have to wait for any request to complete entirely.

    With SPDY we can use request priorities to hint the server which requests should take precedence, even if it arrives later on the wire

    Persistent Connections

    HTTP persistent connection, also called HTTP keep-alive, or HTTP connection reuse,

    is the idea of using a single TCP connection to send and receive multiple HTTP requests/responses,
    as opposed to opening a new connection for every single request/response pair.

    All modern web browsers including Google Chrome, Firefox, Internet Explorer (since 4.01), Opera (since 4.0)[12] and Safari use persistent connections.

    HTTP Persistent Connection

    1. In HTTP/0.9 and 1.0, by default, the connection is closed after a single request/response pair.

      In HTTP 1.0, connections are not considered persistent unless a keep-alive header is included.

      If the client supports keep-alive, it adds an additional header to the request:

      Connection: keep-alive

      Then, when the server receives this request and generates a response, it also adds a header to the response:

      Connection: keep-alive

      Following this, the connection is not dropped, but is instead kept open.
      When the client sends another request, it uses the same connection.
      This will continue until either the client or the server decides that the conversation is over, and one of them drops the connection.

    2. In HTTP 1.1, all connections are considered persistent unless declared otherwise.

      The HTTP persistent connections do not use separate keepalive messages, they just allow multiple requests to use a single connection.

      Advantages of persistent connections:

      1. Reduced latency in subsequent requests (no handshaking).
      2. Reduced CPU usage and round-trips because of fewer new connections and TLS handshakes.
      3. Enables HTTP pipelining of requests and responses.
      4. Reduced network congestion (fewer TCP connections).
      5. Errors can be reported without the penalty of closing the TCP connection.

      As little as one RTT (Time for a small packet to travel from client to server and back) is required for all referenced objects.

      According to RFC 7230, section 6.4, “a client ought to limit the number of simultaneous open connections that it maintains to a given server”.

      HTTP/1.1 specifications stated specific maximum values but in the words of RFC 7230 “this was found to be impractical for many applications… instead… be conservative when opening multiple connections”.

      These guidelines are intended to improve HTTP response times and avoid congestion.

      If HTTP pipelining is correctly implemented, there is no performance benefit to be gained from additional connections, while additional connections may cause issues with congestion.

      Disadvantages
      If the client does not close the connection when all of the data it needs has been received, the resources needed to keep the connection open on the server will be unavailable for other clients.

      How much this affects the server’s availability and how long the resources are unavailable depend on the server’s architecture and configuration.

      Default connection timeout of Apache httpd 1.3 and 2.0 is as little as 15 seconds and just 5 seconds for Apache httpd 2.2 and above.

      The advantage of a short timeout is the ability to deliver multiple components of a web page quickly while not consuming resources to run multiple server processes or threads for too long.

      Keepalive with chunked transfer encoding

      Keepalive makes it difficult for the client to determine where one response ends and the next response begins, particularly during pipelined HTTP operation.
      This is a serious problem when Content-Length cannot be used due to streaming.
      To solve this problem, HTTP 1.1 introduced a chunked transfer coding that defines

      last-chunk

      .
      The last-chunk bit is set at the end of each response so that the client knows where the next response begins.

      Read more on Wikipedia >>>

    3. The newer HTTP/2 protocol uses the same idea and takes it further to allow multiple concurrent requests/responses to be multiplexed over a single connection.

    Chunked transfer encoding is a streaming data transfer mechanism available in HTTP 1.1.

    1. It allows a server to maintain an HTTP persistent connection for dynamically generated content.
      In this case, the HTTP Content-Length header cannot be used to delimit the content and the next HTTP request/response, as the content size is not yet known.
    2. Benefit is that it is not necessary to generate the full content before writing the header, as it allows streaming of content as chunks and explicitly signaling the end of the content, making the connection available for the next HTTP request/response.
    3. Data stream is divided into a series of non-overlapping “chunks”.
    4. Chunks are sent out and received independently of one another.
    5. No knowledge of the data stream outside the currently-being-processed chunk is necessary for both the sender and the receiver at any given time.
    6. Each chunk is preceded by its size in bytes. The transmission ends when a zero-length chunk is received.
    7. The chunked keyword in the Transfer-Encoding header is used to indicate chunked transfer.

    An early form of the chunked transfer encoding was proposed in 1994.

    Chunked transfer encoding is not supported in HTTP/2, which provides its own mechanisms for data streaming.

    Read more on Wikipedia >>>

    1. Long/short polling (client pull)
    2. HTTP/2 Server Push
    3. WebSockets (server push)
    4. Server-Sent Events or SSE (server push)

    Client pull — client asking server for updates at certain regular intervals
    Server push — server is proactively pushing updates to the client (reverse of client pull)
    Read more about push technologies >>>

    Short polling is an AJAX-based timer that calls at fixed delays
    whereas Long polling is based on Comet (i.e server will send data to the client when the server event happens with no delay).

    (Short) Polling (also Ajax Polling):
    Polling is a periodic checking from client (browser) to the server. The server responds immediately with available data. If there is no information empty response will be send to the client. Connection is terminated with or without response.

    Send a request to the server, get an instant answer. Do this every x seconds, minutes etc. to keep your application up-to-date. But: This costs a lot of requests.

    Making repeated requests to server wastes resources as each new incoming connection must be established, the HTTP headers must be passed, a query for new data must be performed, and a response (usually with no new data to offer) must be generated and delivered. The connection must be closed and any resources cleaned up.

    Long polling:
    Send a request to the server, keep the connection open, get an answer when there’s “data” for you. This will cost you only one request (per user), but the request keeps a permanent connection between client and server up.

    HTTP/2 Server Push allows an HTTP/2-compliant server to send resources (“preload”) to a HTTP/2-compliant client before the client requests them.

    HTTP/2 Server Push is not a notification mechanism from server to client.
    It is, for the most part, a performance technique that can be helpful in loading resources preemptively: content is sent even before it is requested.

    Additional resources that the server feels the client would need are also called as promises.

    Push promises are sent as early as possible so that the browser will have them well ahead of any discoveries.

    E.g., the server knows that anyone requesting index.html will need styles.css and script.js,
    so it can push them to the client immediately without waiting for the client to request them.
    If done correctly, by the time the browser finishes parsing index.html, the transfer of styles.css and script.js would have already started, or even completed, removing the latency of having to request these and wait for them to arrive.

    Advantages: This technique is useful while loading web pages which have external style sheets and script files attached to it. Developers don’t need to combine external resources anymore as HTTP/2 optimizes this issue.

    Drawback: this can result in wasted bandwidth if pushed resources go unused by the client.

    How HTTP/2 PUSH works at a protocol level
    Push works over HTTP/2, which at its core is a frames protocol, meaning that information is exchanged in groups of bytes called frames.

    Additionally, frames are part of streams, and streams are identified by a number.
    The stream number is present in each frame as a binary field.
    Streams allow matching requests to responses.

    E.g. the response to request GET /index.html at stream 3 must also be at stream 3.

    Frames Function
    HEADERS frame Carries HTTP headers
    When sent by the browser to the server, it signals that a request is being made.

    When sent by the server to the browser, it signals that a response to a previous request or push promise is being sent.

    PUSH_PROMISE frame Sent by the server to the browser to start pushing a resource.

    It also contains HTTP headers.

    However, the kind of headers present in a PUSH_PROMISE frame are headers that would normally be present in a request. This is different from the response headers that a server would normally send.

    Other headers that may be present in a PUSH_PROMISE and that some browsers use are cache headers, e.g., if-none-match.

    DATA frames Sent in either direction to carry the actual content of a resource or the contents that the browser POSTs or PUTs to the server.
    RST_STREAM frames These frames serve many purposes.
    One of them is having the browser signal to the server that a pushed stream is not needed.
    1. When the server wants to push a resource, it prepares a PUSH_PROMISE frame, architecting it in the best way possible to seduce the browser into using the pushed contents.
    2. Then the server annexes the PUSH_PROMISE frame to the response part of a normal, browser-initiated stream.
    3. However, the actual data for the pushed resource is sent in a fresh stream started by the server – and thus with an even number.
    4. The browser holds the pushed data in a temporary “quarantine” zone until it decides to use it.
    5. Later, every time the browser is going to make an actual request, it examines the contents of any received push promises to see if it is similar enough to the request it wants to make.
    6. However, the server need not wait until that moment to start sending data for the promised resource.
    7. After the PUSH_PROMISE frame has been sent on the browser-initiated stream, the server can send what would be response headers using a HEADERS frame in the new server-initiated stream, and later it can send the data of the resource using DATA frames.
    8. And at any point in time, the browser can interrupt any transfer by using RST_STREAM.

    E.g., the server knows that anyone requesting index.html will need styles.css and script.js

    1. Server receives HEADERS frame asking for index.html in stream 3, and it can forecast the need for styles.css and script.js.
    2. Server sends a PUSH_PROMISE for styles.css and a PUSH_PROMISE for script.js, again in stream 3. These frames are roughly equivalent to a browser’s request.
    3. Server sends a HEADERS frame in stream 3 for responding to the request for index.html.
    4. Server sends DATA frame(s) with the contents of index.html, still in stream 3.
    5. Server sends HEADERS frame for the response to styles.css in stream 4 – notice the even stream number – and then for the response to script.js in stream 6.
    6. Server sends DATA frames for the contents of styles.css and script.js, using their respective stream numbers.

    Read more on Wikipedia >>>

    WebSocket

    WebSocket is a computer communications protocol, providing full-duplex communication channels over a single TCP connection.

    Supported Browsers
    WebSocket is supported by most browsers: Google Chrome, Microsoft Edge, Internet Explorer, Firefox, Safari and Opera.

    History of WebSocket:

    1. WebSocket was first referenced as TCPConnection in the HTML5 specification, as a placeholder for a TCP-based socket API.
    2. In June 2008, a series of discussions were led by Michael Carter that resulted in the first version of the protocol known as WebSocket.
    3. The WebSocket protocol was standardized by the IETF as RFC 6455 in 2011, and the WebSocket API in Web IDL is being standardized by the W3C.
    WebSocket vs. HTTP

    Similarities:

    1. Both protocols are located at layer 7 in the OSI model (TCP/IP Layer 5) and depend on TCP at layer 4.
    2. Although they are different, RFC 6455 states that WebSocket
      • “is designed to work over HTTP ports 80 and 443
      • as well as to support HTTP proxies and intermediaries,”

      thus making it compatible with the HTTP protocol.

    3. To achieve compatibility, the WebSocket handshake uses the HTTP Upgrade header to change from the HTTP protocol to the WebSocket protocol.

    Distinctions:

    1. WebSocket is distinct from HTTP.
    2. Unlike HTTP, WebSocket provides full-duplex communication.
    3. The WebSocket enables client-server interaction
      with lower overhead than half-duplex alternatives such as HTTP polling
      (eliminates the need for the long polling paradigm),
      facilitating real-time data transfer from and to the server.
    4. This is made possible by:
      1. providing a standardized way for the server to send content to the client without being first requested by the client
      2. and allowing messages to be passed back and forth while keeping the connection open.
    5. In this way, a two-way ongoing conversation can take place between the client and the server.
    6. The communications are done over TCP port number 80
      (or 443 in the case of TLS-encrypted connections),
      which is of benefit for those environments which block non-web Internet connections using a firewall.

    Downfalls of WebSockets:

    1. With WebSockets, we need to handle lots of problems that are taken care of in HTTP on their own.
    2. WebSocket is a different protocol for delivering data, it’s not automatically multiplexed over HTTP/2 connections.
    3. Implementing custom multiplexing both on the server and the client is bit complicated.
    4. WebSockets are frame-based and not stream-based. When we open the network tab. You can see the WebSocket messages are listed underframes.

    Read more on Wikipedia >>>

    Read MDN doc that explains WebSocket in detail >>>

    WebSocket vs. Comet

    Similar two-way browser-server communications
    have been achieved in non-standardized ways
    using stopgap technologies such as Comet.

    Before WebSocket, port 80 full-duplex communication was attainable using Comet channels.

    Comet is a web application model in which a long-held HTTPS request allows a web server to push data to a browser, without the browser explicitly requesting it.

    Comet is an umbrella term, encompassing multiple techniques for achieving this interaction.

    All these methods rely on features included by default in browsers, such as JavaScript, rather than on non-default plugins.

    The use of Comet techniques in web development predates the use of the word Comet as a neologism for the collective techniques.

    The term Comet is not an acronym, but was coined by Alex Russell in his 2006 blog post Comet: Low Latency Data for the Browser.

    Comet is known by several other names, including:

    1. Ajax Push
    2. Reverse Ajax
    3. Two-way-web
    4. HTTP Streaming
    5. and HTTP server push among others.

    Comet Implementations

    Specific methods of implementing Comet fall into two major categories: streaming and long polling.

    1. Streaming
      1. Hidden iframe
        1. Hidden iframe (inline frame) HTML element allows a website to embed one HTML document inside another.
        2. It is sent as a chunked block, which implicitly declares it as infinitely long (sometimes called “forever frame”).
        3. As events occur, the iframe is gradually filled with script tags, containing JavaScript to be executed in the browser.
        4. Because browsers render HTML pages incrementally, each script tag is executed as it is received.
        5. One benefit of the iframes method is that it works in every common browser.
        6. Two downsides of this technique are the lack of a reliable error handling method, and the impossibility of tracking the state of the request calling process.
      2. XMLHttpRequest
        1. The XMLHttpRequest (XHR) object is a tool used by Ajax applications for browser–server communication.
        2. It can also be pressed into service for server–browser Comet messaging by
          1. generating a custom data format for an XHR response
          2. and parsing out each event using browser-side JavaScript
        3. It relies only on the browser firing the onreadystatechange callback each time it receives new data.
    2. Ajax with Long Polling
      1. The HTTP protocol is based on a request/response pattern, which means that the server cannot push any data to the client.
        The server can only provide data to the client in response to a client request).
      2. Long polling is a web application development pattern used to emulate pushing data from server to client.
      3. Client polls the server for an event (or set of events).
      4. browser makes an Ajax-style request to the server
      5. Connection remains open until the server has new data to send to the browser.
      6. Data is sent to the browser in a complete response.
      7. The connection is closed only after data is sent back to the client or connection timeout occurs.
      8. The client then creates a new request when the connection is closed, thus restarting the loop.

      Drawback of using long Polling

      1. Long polling requests are not different from any other HTTP request and web servers handle them the same way.

        This means that every long poll connection will reserve server resources, potentially maxing out the number of connections the server can handle.
        This can lead to HTTP connection timeouts.

      2. Each web browser will limit the maximum number of connections web application can make.
        This means that your application load time and performance may be degraded.
      3. Message ordering and delivery guarantees.
        Message ordering cannot be guaranteed if the same client opens multiple connections to the server.
        If the client was not able to receive the message then there will be possible message loss.

      Long Polling Technologies:

      1. XMLHttpRequest Long Polling
        1. For the most part, XMLHttpRequest long polling works like any standard use of XHR.
        2. The browser makes an asynchronous request of the server, which may wait for data to be available before responding.
        3. The response can contain encoded data (typically XML or JSON) or Javascript to be executed by the client.
        4. At the end of the processing of the response, the browser creates and sends another XHR, to await the next event.
        5. Thus the browser always keeps a request outstanding with the server, to be answered as each event occurs.
      2. Script Tag Long Polling
        1. While any Comet transport can be made to work across subdomains, none of the above transports can be used across different second-level domains (SLDs), due to browser security policies designed to prevent cross-site scripting attacks.
        2. Unlike iframes or XMLHttpRequest objects, script tags can be pointed at any URI, and JavaScript code in the response will be executed in the current HTML document.
        3. A long-polling Comet transport can be created by dynamically creating script elements, and setting their source to the location of the Comet server, which then sends back JavaScript (or JSONP) with some event as its payload.
        4. Each time the script request is completed, the browser opens a new one, just as in the XHR long polling case.
        5. This method has the advantage of being cross-browser while still allowing cross-domain implementations.

    However, Comet implementation is nontrivial,
    and due to the TCP handshake and HTTP header overhead,
    it is inefficient for small messages.

    The WebSocket protocol aims to solve these problems without compromising security assumptions of the web.

    HTML5 alternatives to long polling

    1. WebSocket
    2. Server-sent DOM Events

    Read more about Comet on Wikipedia >>>

    Server-Sent Events
    1. Unlike WebSockets, Server-Sent Events are a one-way communication channel where events flow from server to client only.
    2. Server-Sent Events allows browser clients to receive a stream of events from a server over an HTTP connection without polling (without repeatedly requesting it).
    3. Browser client creates a connection using an EventSource API with a server endpoint which is expected to return a stream of events over time. This essentially makes an HTTP request at given URL.
    4. The server receives a regular HTTP request from the client and opens the connection and keeps it open. The server can now send the event data as long as it wants or it can close the connection if there are no data.
    5. Server asynchronously pushes the data to the client once the client-server connection is established.
    6. It can be considered as a one-way publish-subscribe model.
    7. A client subscribes to a “stream” from a server and the server will send messages (“event-stream”) to the client until the server or the client closes the stream.
    8. It is up to the server to decide when and what to send the client, e.g. as soon as data changes. The server can decide to send data whenever a new “chunk” of data is available.
    9. The client receives each event from the server and process it. If it receives a close signal from the server it can close the connection. The client can also initiate the connection close request.
    10. SSE offers a standard JavaScript client API named EventSource and a new MIME type text/event-stream implemented in most modern browsers as part of the HTML5 standard by W3C.
      Polyfills are available for browsers that do not support EventSource API.
    11. All major browsers except Microsoft Edge include this technology.

    Main Benefits of SSE:

    1. As SSE is based on HTTP, it is more compliant with existing IT infrastructure like (Load Balancer, Firewall, etc), unlike WebSockets which can be blocked by some firewall.
    2. Simpler implementation and Data efficiency
    3. It is automatically multiplexed over HTTP/2 out of the box
    4. Limits the number of connections for data on the client to one

    Read more about SSE on Wikipedia >>>

    Which Pushing Approach to choose?

    Depends on the situation:

    1. MMO (Massive Multiplayer Online) Games that need a huge amount of messages from both ends of the connection are best implemented using WebSockets.
    2. In case of displaying real-time market news, market data, chat applications, etc., relying on HTTP/2 + SSE will provide you with an efficient bi-directional communication channel while reaping the benefits from staying in the HTTP world.

    HTTP 1.1 vs. HTTP/2
    1. HTTP/2 provides decreased latency to improve page load speed by supporting:
      1. Data compression of HTTP headers
      2. HTTP/2 Server Push
      3. Prioritization of requests
      4. Multiplexing multiple requests over a single TCP connection
      5. An important operational benefit of HTTP/2 is that it avoids the head-of-line blocking problem in HTTP 1.
    2. HTTP/2 is a binary protocol based on SPDY.
      The biggest difference between HTTP/1.1 and SPDY was that very HTTP/2 request and response is given a unique id called “stream ID”, meaning there is a single TCP channel connecting the user to the server.

      1. A stream is a collection of frames with a same stream id.
      2. Client divides the request into frames (binary pieces of data) and assigns the stream ID of the request to the frames.
      3. Then it initiates a TCP connection with the server and starts sending frames to the server.
      4. Once the server has the response ready it divides the response into frames and gives the response frames the same response stream id.
      5. Multiple requests to an origin is made using a single TCP connection, so
        Stream id is used to identify to which request or response a frame belongs to.
    3. Since HTTP/2 runs on top of a single TCP connection there is still potential for head-of-line blocking to occur if TCP packets are lost or delayed in transmission.
    4. HTTP/2 no longer supports HTTP 1.1’s chunked transfer encoding mechanism, as it provides its own, more efficient, mechanisms for data streaming.

    The development of HTTP/2 used SPDY as a jumping-off point.

    Among the many detailed differences between the protocols, the most notable is that HTTP/2 uses a fixed Huffman code-based header compression algorithm, instead of SPDY’s dynamic stream-based compression. This helps to reduce the potential for compression oracle attacks on the protocol, such as the CRIME attack.

    On February 9, 2015, Google announced plans to remove support for SPDY in Chrome in favor of support for HTTP/2.[27] That took effect, starting with Chrome 51.

    1. Headers in HTTP/2 provide information about the request or response.
    2. Header take around 800 bytes of bandwidth and sometimes few KB if it carries cookies.
    3. HTTP/2 introduced header compression to reduce the bandwidth latency.
    4. Header compression is not like request/response body gzip compression.
      It is a technique of not sending the same headers again.
    5. For every TCP connection the client and server maintain a headers table containing the last response and request headers and their values respectively.
    6. For the first request or response they send all the required header.
      But for subsequent requests client and server omit headers which are same as the previous request or response.
    7. Those “repeated” headers are retrieved from the headers table thus decreasing the overall length of the headers data in a request or response.
    8. All the header fields of HTTP/1.1 remains the same in HTTP/2. Just the first line of the HTTP/1.1 request is split into key-value pairs i.e., :method, :scheme, :host and :path.
    Why do we need header compression?

    As per HTTP/2 Q&As:
    “Patrick McManus from Mozilla showed this vividly by calculating the effect of headers for an average page load.

    If you assume that a page has about 80 assets (which is conservative in today’s Web), and each request has 1400 bytes of headers (again, not uncommon, thanks to Cookies, Referer, etc.), it takes at least 7-8 round trips to get the headers out “on the wire.” That’s not counting response time – that’s just to get them out of the client.

    This is because of TCP’s Slow Start mechanism, which paces packets out on new connections based on how many packets have been acknowledged – effectively limiting the number of packets that can be sent for the first few round trips.

    In comparison, even mild compression on headers allows those requests to get onto the wire within one roundtrip – perhaps even one packet.

    This overhead is considerable, especially when you consider the impact upon mobile clients, which typically see round-trip latency of several hundred milliseconds, even under good conditions.”

    HPACK vs. GZIP

    A regular HTTPS connection is an overlay of several connections in the multi-layer model.
    The most basic connection you usually care about is the TCP connection (the transport layer),
    on top of that you have the TLS connection (mix of transport/application layers),
    and finally the HTTP connection (application layer).

    In the the old days, HTTP compression was performed in the TLS layer, using GZIP.
    Both headers and body were compressed indiscriminately, because the lower TLS layer was unaware of the transferred data type.
    In practice it meant both were compressed with the DEFLATE algorithm.

    SPDY/2 proposed using a single GZIP context in each direction for header compression, which was simple to implement as well as efficient.

    Although specifically designed for headers, including the use of a preset dictionary, it was still using DEFLATE, including dynamic Huffman codes and string matching.

    Since then, a major attack has been documented against the use of stream compression (like GZIP) inside of encryption; CRIME.

    With CRIME, it’s possible for an attacker who has the ability to inject data into the encrypted stream to “probe” the plaintext and recover it.

    Because DEFLATE uses backward string matches and dynamic Huffman codes, an attacker that can control part of the request headers, can gradually recover the full cookie by modifying parts of the request and seeing how the total size of the request changes during compression.

    JavaScript makes this possible, and there were demonstrations of recovery of cookies and authentication tokens using CRIME for TLS-protected HTTP resources.

    Most edge networks, including Cloudflare, disabled header compression because of CRIME. That’s until HTTP/2 came along.

    As a result, HTTP/2 could not use GZIP compression. A new, header-specific compression scheme that operates at a coarse granularity, was created for HTTP/2.

    HTTP/2 supports a new dedicated header compression algorithm, called HPACK. HPACK was developed with attacks like CRIME in mind, and is therefore considered safe to use.

    Although HPACK does string matching, for the attacker to find the value of a header, they must guess the entire value, instead of a gradual approach that was possible with DEFLATE matching, and was vulnerable to CRIME.

    HPACK uses these three methods of compression:

    1. Static Dictionary: A predefined dictionary of 61 commonly used header fields, some with predefined values.
    2. Dynamic Dictionary: A list of actual headers that were encountered during the connection. This dictionary has limited size, and when new entries are added, old entries might be evicted.
    3. Huffman Encoding: A static Huffman code can be used to encode any string: name or value. This code was computed specifically for HTTP Response/Request headers – ASCII digits and lowercase letters are given shorter encodings. The shortest encoding possible is 5 bits long, therefore the highest compression ratio achievable is 8:5 (or 37.5% smaller).
    HPACK Flow

    When HPACK needs to encode a header in the format name:value, it will first look in the static and dynamic dictionaries. If the full name:value is present, it will simply reference the entry in the dictionary. This will usually take one byte, and in most cases two bytes will suffice!

    Since many headers are repetitive, this strategy has a very high success rate.

    When HPACK can’t match a whole header in a dictionary, it will attempt to find a header with the same name. Most of the popular header names are present in the static table, for example: content-encoding, cookie, etag. The rest are likely to be repetitive and therefore present in the dynamic table.

    If the name was found, it can again be expressed in one or two bytes in most cases, otherwise the name will be encoded using either raw encoding or the Huffman encoding: the shorter of the two. The same goes for the value of the header.

    Read more about HPACK >>>

    Users, Groups, Permissions

    /etc/passwd is the primary file in Unix/Linux operating system that stores information about user accounts and can be read by all users.

    /etc/shadow many times is used by the operating system instead due to security concerns and increased hashing capabilities.

    /etc/shadow more often than not is highly restricted to privileged users.

    If you as a Linux admin “What is root”, you may very well get the response “root, god, what’s the difference?”
    Essentially root is THE admin, but in a Linux environment it is important to remember that unlike in a Windows environment, you spend very little time in a “privileged” mode.

    Many Windows programs over the years have required that the user be a local admin in order to function properly and have caused huge security issues as a result.

    This has changed some over the years, but it can still be difficult to remove all of the programs asking for top level permissions.

    A Linux user remains as a standard user nearly all the time, and only when necessary do they change their permissions to that of root or the superuser (su). sudo (literally- superuser do …) is the main way used to run one-off commands as root, or it is also possible to temporarily have a root-level bash prompt.

    UAC (User Account Control) is similar in theme to sudo, and like Windows Firewall can be a pain in the neck but it does do a lot of good. Both programs allow the user to engage higher-level permissions without having to log out of their current user session- a massive time saver.

    Giving a user as few privileges as possible tends to cause some aggravation by the user, but by the same token it also removes a lot of easily preventable infection vectors. Still, sometimes users need to have local admin rights in order to troubleshoot issues- especially if they’re on the road with a laptop. Therefore, creating a local admin account may sometimes be the most effective way to keep these privileges separate.

    Error 5 is very common when dealing with files and directories that have very specific permissions. When trying to copy elements from areas that have restricted permissions, or when trying to copy files to an area that has restricted permissions, you may get this error which basically means “Access denied”. Checking out permissions, making sure that you have the appropriate permissions to both the source and destination locations, and making yourself the owner of those files can help to resolve this issue. Just remember that if you are not intended to be able to view these files to return the permissions back to normal once you are finished.

    Unix/Linux permissions operate on much simpler methodology than Windows does.
    Linux permissions are normally visible using the following scale:

    d | rwx | rwx | rwx. 
    

    This stretch of characters actually represents four distinct sections of binary switches:

    1. directory

      The first value (d)- asks ‘is this a directory’

    2. owner

      represents what permissions the owner of the file has for read (r), write (w), and execute (x)

    3. group

      represent what members of the group can do for read, write and execute

    4. other

      what everybody else can do

    E.g., when reading permissions with the value of

    drwxr-xr-x
    

    it would mean that it is a directory,
    the owner has full permissions,
    while everybody else can read and execute, nobody else can write to it.

    Looking at this as a basic yes/no (1/0) system, we would have binary values for each of these fields: 1 for yes, 0 for no.

    Binary to numbers mapping:

    • 0000 = 0
    • 0001 = 1
    • 0010 = 2
    • 0011 = 3
    • 0100 = 4
    • 0101 = 5
    • 0110 = 6
    • 0111 = 7
    rwx rwx rwx
    111 101 101
    7 5 5

    This would give us 755 as shorthand for owner read, write and execute, and everybody else is read and execute.

    rwx rwx rwx
    110 100 100
    4 6 6

    This would give us 644 as shorthand for owner read and write, with everybody else read-only: rw-r–r–

    If you try to work out permissions for every single person in your organization individually you can give yourself a migraine pretty quickly.

    Therefore, trying to simplify permissions but keep them strong is critical to administering a large network.

    Groups allow users to be pooled by their need to know and need to access particular information.

    In this way, it allows the administrator to set the permissions once- for the group- then add users to that group.

    When modifications to permissions need to be made, its one change that affects all members of that group.

    Security

    A Firewall put simply keeps stuff from here talking to stuff over there.

    Firewalls exist in many different possible configurations with both hardware and software options as well as network and host varieties.

    Most of the general user base had their first introduction to Firewalls when Windows XP SP2 came along with Windows Firewall installed. This came with a lot of headaches, but to Microsoft’s credit it did a lot of good things.

    Over the years it has improved a great deal and while there are still many options that go above and beyond what it does, what Windows Firewall accomplishes it does very well. Enhanced server-grade versions have been released as well, and have a great deal of customization available to the admin.

    Difference between host firewalls and network firewalls

    A host firewall runs on a computer, or server, to block or allow incoming connections to specific local services. The goal of such firewall is to protect the host itself where the firewall is running. Most modern operating systems, like Windows, Mac, or Linux, have available such a service.

    A network firewall is a dedicated device that is installed on a specific network segment to protect one or more private networks that reside behind it. Network firewalls are sophisticated appliances that can inspect a large amount of throughput data adding little delay.

    Difference between reject and drop in firewall configurations

    Firewalls can block connections via two methods: reject or drop. When a connection is rejected, the firewall tells the source that the destination host is not allowing incoming connections to specific port(s).

    The second method is to silently drop the packets, acting as if the host is unreachable. As I described earlier, since this method is more difficult to troubleshoot, it’s more appropriate to slow down hostile users that are scanning a network in the hopes of finding vulnerabilities to exploit. For this reason, it’s a good idea to configure internal firewalls with reject rules and external ones, while facing public networks with drop rules.

    What is DNS spoofing?
    1. DNS spoofing is a type of computer attack where user is forced to navigate to a fake website that looks like a real one.
    2. Its intention is to diverting traffic and to get user data.

    There are currently two methods of establishing a secure HTTP connection:

    1. HTTP 1.1 Upgrade Header, introduced by RFC 2817.

      Browser support for the Upgrade header is, however, nearly non-existent, so HTTPS is still the dominant method of establishing a secure HTTP connection.

    2. HTTPS URI scheme
      1. HTTPS is a secure version of HTTP.
      2. HTTPS is notated by the prefix https:// instead of http:// on web URIs.
      3. In this protocol, data transfer over the World Wide Web is secure and encrypted.
      4. It is used to execute highly confidential online transaction like financial transactions.

    HTTP HTTPS
    (HTTP is a protocol using which hypertext is transferred over the Web. Due to its simplicity, http has been the most widely used protocol for data transfer over the Web Think of ‘Online shopping’ at Amazon or Flipkart. You might have noticed that as soon as we click on the Check-out on these online shopping portals, the address bar gets changed to use https. This is done so that the subsequent data transfer (i.e. financial transaction etc.) is made secure. And that’s why https was introduced so that a secure session is setup first between Server and Browser
    http isn’t as secure as we would like it to be.
    In fact, hyper-text exchanged using http goes as plain text i.e. anyone between the browser and server can read it relatively easy if one intercepts this exchange of data.
    cryptographic protocols such as SSL and/or TLS turn http into https i.e. https = http + cryptographic protocols. Also, to achieve this security in https, Public Key Infrastructure (PKI) is used because public keys can be used by several Web Browsers while private key can be used by the Web Server of that particular website. The distribution of these public keys is done via Certificates which are maintained by the Browser. You can check these certificates in your Browser settings.
    Default Port http uses default port 80 https uses default port 443
    Speed Faster Slower
    security in https is achieved at the cost of processing time because Web Server and Web Browser needs to exchange encryption keys using Certificates before actual data can be transferred. Basically, setting up of a secure session is done before the actual hypertext exchange between server and browser.
    HTTP Works at Application Layer HTTPS works at Transport Layer
    In HTTP, Encryption is absent Encryption is present in HTTPS
    HTTP does not require any certificates HTTPS needs SSL Certificates

    The major benefits of HTTPS certificate are:

    1. Customer information like credit card number and ATM pin is encrypted and cannot be easily track.
    2. Customers trust and prefer to purchase from the sites that use HTTPS protocol.
    3. This protocol shows authenticate register domain as secure connection.

    1. HTTPS is a URI scheme that is, aside from the scheme token, syntactically identical to the http scheme used for normal HTTP connections, but which signals the browser to use an added encryption layer of SSL/TLS to protect the traffic.
    2. SSL is especially suited for HTTP since it can provide some protection even if only one side of the communication is authenticated. This is the case with HTTP transactions over the Internet, where typically only the server is authenticated.

    The three basic ways to authenticate someone are: something they know (password), something they have (token), and something they are (biometrics). Two-factor authentication is a combination of two of these methods, oftentimes using a password and token setup, although in some cases this can be a PIN and thumbprint.

    An IDS is an Intrusion Detection System with two basic variations: Host Intrusion Detection Systems and Network Intrusion Detection Systems. An HIDS runs as a background utility in the same as an anti-virus program for instance, while a Network Intrusion Detection System sniffs packets as they go across the network looking for things that aren’t quite ordinary. Both systems have two basic variants- signature based and anomaly based. Signature based is very much like an anti-virus system, looking for known values of known ‘bad things’ while anomaly looks more for network traffic that doesn’t fit the usual pattern of the network. This requires a bit more time to get a good baseline, but in the long term can be better on the uptake for custom attacks.

    As per Wikipedia:

    1. Use a minimum password length of 12 to 14 characters if permitted.
    2. Include lowercase and uppercase alphabetic characters, numbers and symbols if permitted.
    3. Generate passwords randomly where feasible.
    4. Avoid using the same password twice (eg. across multiple user accounts and/or software systems).
    5. Avoid character repetition, keyboard patterns, dictionary words, letter or number sequences, usernames, relative or pet names, romantic links (current or past) and biographical information (e.g. ID numbers, ancestors’ names or dates).
    6. Avoid using information that is or might become publicly associated with the user or the account.
    7. Avoid using information that the user’s colleagues and/or acquaintances might know to be associated with the user.
    8. Do not use passwords which consist wholly of any simple combination of the aforementioned weak components.

    Session, Connections, Caching

    1. Similar to how a DNS server caches the addresses of accessed websites,
      a proxy server caches the contents of those websites and handles the heavy lifting of access and retrieval for users.
    2. Proxy servers can also maintain a list of blacklisted and whitelisted websites so as to prevent users from getting easily preventable infections.
    3. Depending on the intentions of the company, Proxy servers can also be used for monitoring web activity by users to make sure that sensitive information is not leaving the building.
    4. Proxy servers also exist as Web Proxy servers, allowing users to either not reveal their true access point to websites they are accessing and/or getting around region blocking.

    1. HTTP session is a sequence of network request-response transactions.
    2. An HTTP client initiates a request.
    3. It establishes a Transmission Control Protocol (TCP) connection to a particular port on a host (typically port 80; see List of TCP and UDP port numbers).
    4. An HTTP server listening on that port waits for a client’s request message.
    5. Upon receiving the request, the server sends back
      1. a status line, such as “HTTP/1.1 200 OK”
      2. and a message of its own, the body of which is
        1. the requested resource
        2. error message
        3. or some other information.

    1. HTTP is a stateless protocol.
    2. A stateless protocol does not require the server to retain information or status about each user for the duration of multiple requests.
    3. E.g., when a web server is required to customize the content of a web page for a user,
      the web application may have to track the user’s progress from page to page.
    4. A common solution is the use of HTTP cookies.
    5. Other methods include:
      1. server side sessions
      2. hidden variables
        (when the current page is a form)
      3. URL-rewriting using URI-encoded parameters,
        e.g., /index.php?session_id=some_unique_session_code.

      Difference between stateless and stateful protocols
    1. A stateless communications protocol treats each request as an independent transaction.
      It therefore does not require the server to retain any session, identity, or status information spanning multiple requests from the same source. Similarly, the requestor can not rely on any such information being retained by the responder.
    2. In contrast, a stateful communications protocol is one in which the responder maintains “state” information (session data, identity, status, etc.) across multiple requests from the same source.
    Which type of protocol is HTTP?
    1. HTTP is a stateless protocol.
      HTTP does not require server to retain information or status about each user for the duration of multiple requests.
    2. Some web servers implement states using different methods (using cookies, custom headers, hidden form fields etc.).
    3. However, in the very core of every web application everything relies on HTTP which is still a stateless protocol that is based on simple request/response paradigm.

    A server session creates a file in a temporary directory on the server where registered session variables and their values are stored.

    This data will be available to all pages on the site during that visit.

    A session ends when the user loses the browser or after leaving the site, the server will terminate the session after a predetermined period of time, commonly 30 minutes duration.

    The browser session remains as long as the user has kept the browser opened.

    As soon as he closes it, the session expires.

    And once this session expires, it sends data to the server to close this session over there as well.

    Session-based authentication (which maintains server state)
  • On login, server generates a unique (very difficult to guess) session id, passes this to the client (browser) which stores it (eg. browser cookie)
  • Maintains all session state on the server keyed on session id
  • All future incoming requests from client provide this session id to server, which can authenticate the client
  • Token-based authentication (which is stateless)
    1. On login, server generates an authentication token based on some cryptographic scheme, passes this to the client (browser) which stores it (eg. browser cookie)
    2. No state is maintained on the server
    3. All future incoming requests from client provide the auth token to server. Server then uses the features of the cryptographic scheme to verify the auth token on the fly to authenticate client

    Cookies and sessions are both ways to preserve the application’s state between different requests the browser makes. It’s thanks to them that, for instance, you don’t need to log in every time you request a certain web page.

    Purpose
    Session is used for maintaining a dialogue between server and user.
    Cookies maintain user identification by tracking visitor record.

    Reset/expiration time
    Session values are reset
    after a period of time has passed without a connection from the client with which the session is associated.
    E.g., data in the session is lost when the web browser is closed.

    Cookies values are normally saved
    until their expiry time (or the browser is closed if one isn’t set).
    Data stored in cookie can be stored for months or years, depending on the life span of the cookie.

    E.g., setting expiry datetime using HTTP headers:

    Set-Cookie: name2=value2; Expires=Wed, 19 Jun 2021 10:18:14 GMT

    Expire immediately overrides are available for both.

    Where stored
    The main difference between a session and a cookie is that session data is stored on the server, whereas cookies store data in the visitor’s browser.

    Amount of data
    A cookie stores limited amount of data, up to 4kb[4096bytes].
    We can have more than one cookie.
    A single Cookie can not hold multiple values

    There is no such storage limit on session.
    Sessions can hold multiple variables.

    Security and controlled by
    Sessions are more secure than cookies as it is stored in server. Cookie can be turned off from browser.

    Can you store a cookie in a session?
    If a piece of data is stored on the server, then it isn’t being stored by the browser (copies of data not withstanding), so you can’t store a cookie in a session.

    Cookies
    1. A cookie is a snippet of data sent to and returned from clients.
    2. Cookies are small bits of data, (maximum of 4KB long), which hold data in a key=value pairs:
      name=value; name2=value2

      These are set either by JavaScript, or via the server using an HTTP header.

    3. Cookies are usually used to preserve login state, where a username and a special hash are sent from the browser, and the server checks them against the database to approve access.
    4. Cookies are also used to identify sessions.
      In any site that is using cookies you can see that there is a header sent to a server and also received called Cookie.
      Usually it contains some personal information (e.g., ID) that can be used on the server to identify a session.
      These cookies stay on your computer and your browser takes care of sending them to only the domains that are identified with it.

      If there were no cookies then you would be sending a unique ID on every request via GET or POST. Cookies are like static id’s that stay on your computer for some time.

    5. Cookies are considered highly insecure because the user can easily manipulate their content. That’s why you should always validate cookie data. Don’t assume what you get from a cookie is necessarily what you expect.
    Sessions
    1. A session is a collection of data stored on the server and associated with a given user
    2. Information on the server that is associated with the cookie information can be seen in PHP by checking the session.save_path location and actually “see sessions”. They are either files on the server filesystem or backed in a database.
    3. Session maintains state between HTTP requests. HTTP is fundamentally a stateless protocol; sessions are used to give it statefulness.
    4. Each user gets a session ID, which is sent back to the server for validation either by cookie (containing an id code) or by GET variable.
    5. Session IDs transmitted via GET variable would mean that the session_id would need to be appended to every page (hello.php?sid=cbe709ac7bed98f7ecb89713).
    6. Sessions are usually short-lived, which makes them ideal in saving temporary state between applications.
    7. Sessions also expire once the user closes the browser.
    8. Sessions are considered more secure than cookies because the variables themselves are kept on the server.

    Here’s how Session works:

    1. Server opens a session (sets a cookie via HTTP header)
    2. Server sets a session variable.
    3. Client changes page
    4. Client sends all cookies, along with the session ID from step 1.
    5. Server reads session ID from cookie.
    6. Server matches session ID from a list in a database (or memory etc).
    7. Server finds a match, reads variables which are now available on $_SESSION superglobal.
    8. If PHP does not find a match, it will start a new session, and repeat the steps from 1-7.

    You can store sensitive information on a session because it is kept on the server.

    Session ID can still be stolen if the user logged in over an insecure WiFi.
    An attacker can sniff the cookies, and set it as their own, he won’t see the variables themselves, but the server will identify the attacker as the user.

    Caching is a practice of:

    • temporarily storing the data
    • and retrieving data from a high-performance store (usually memory) either explicitly or implicitly.

    When a caching mechanism is in place, it improves delivery speed by:

    1. storing a copy of the asset you requested
    2. and later accessing the cached copy instead of the original.

    Web cache or also known as HTTP cache is a temporary storage where HTML pages and images are stored temporarily so that server lag could be reduced.

    1. Web cache has its own disk space
    2. Web cache can act both like server and client
    3. Web cache might reduce the response time
    4. Web cache contains copies of recently requested objects

    Cacheable response is an HTTP response that can be cached,
    that is stored to be retrieved and used later,
    saving a new request to the server.

    Not all HTTP responses can be cached,
    there are the following Constraints for an HTTP response to be cached:

    1. Cacheable Request Method
      The method used in the request is itself cacheable,
    2. Cacheable response status code:
      The status code of the response is known by the application caching, and it is considered cacheable.
    3. No specific headers in the response are used (e.g., Cache-Control) to prevents caching.

    E.g., when both, the method of the request and the status of the response, are cacheable, the response to the request can be cached:

    GET /pageX.html HTTP/1.1
    (…) 
    
    200 OK
    (…)
    

    Read more on MDN >>>

    Request methods can be defined as “cacheable
    to indicate that
    responses to them are allowed to be stored for future reuse;
    for specific requirements see [RFC7234].

    In general, safe methods that do not depend on a current or authoritative response are defined as cacheable.

    1. Cacheable Methods:
      This specification defines GET, HEAD, and POST as cacheable,
      although the overwhelming majority of cache implementations only support GET and HEAD.
    2. Can be Cacheable:
      a response to a POST or PATCH request can also be cached if freshness is indicated and the Content-Location header is set, but this is rarely implemented.
      (E.g.,, Firefox does not support it per https://bugzilla.mozilla.org/show_bug.cgi?id=109553.)
    3. Not Cacheable Methods:
      other methods, like PUT or DELETE are not cacheable and their result cannot be cached.

    Read more on IETF >>

    The following status code are cacheable:

    1. 200, 203, 204, 206,
    2. 300, 301,
    3. 404, 405, 410, 414,
    4. 501.

    Read more about Response Status Codes >>>

    Some non-cacheable requests/responses to a specific URI may invalidate previously cached responses on the same URI.

    E.g., A PUT request cannot be cached. Moreover, a PUT to pageX.html will invalidate all cached GET or HEAD requests to the same URI.

    PUT /pageX.html HTTP/1.1
    (…)
    
    200 OK
    (…)
    

    A specific Cache-Control header in the response can prevent caching:

    GET /pageX.html HTTP/1.1
    (…)
    
    200 OK
    Cache-Control: no-cache
    (…)
    

    The conditional GET mechanism helps to keep a cache up to date.

    The HTTP protocol requests the server of the website its trying to access so that it can store its files, images etc. in cache memory.

    This request of asking the server for a document considering a specific parameter is called conditional GET Request.

    1. An ETag is an opaque identifier assigned by a web server to a specific version of a resource found at an URL.
    2. If the resource content at that URL ever changes, a new and different ETag is assigned.
    3. In typical usage, when an URL is retrieved the web server will return the resource along with its corresponding ETag value, which is placed in an HTTP “ETag” field:
      ETag: "unique_id_of_resource_version"
    4. The client may then decide to cache the resource, along with its ETag.
    5. Later, if the client wants to retrieve the same URL again, it will send its previously saved copy of the ETag along with the request in a “If-None-Match” field.
      If-None-Match: "unique_id_of_resource_version"
    6. On this subsequent request, the server may now compare the client’s ETag with the ETag for the current version of the resource.
    7. If the ETag values match, meaning that the resource has not changed, then the server may send back a very short response with a HTTP 304 Not Modified status.
      The 304 status tells the client that its cached version is still good and that it should use that.
    8. However, if the ETag values do not match, meaning the resource has likely changed, then a full response including the resource’s content is returned, just as if ETag were not being used.
      In this case the client may decide to replace its previously cached version with the newly returned resource and the new ETag.

    Request methods in HTTP

    1. HTTP verbs tell the server what to do with the data identified by the URL.
    2. The request can optionally contain additional information in its body, which might be required to perform the operation – for instance, data you want to store with the resource. You can supply this data in cURL with the -d option.

    Request methods in HTTP
    Method Description
    GET
    1. GET is the simplest type of HTTP request method;
      Browsers use it each time you click a link or type a URL into the address bar.
    2. It is used to send data in url.
    3. Get requests a representation of the specified resource to retrieve data from the server.
    4. It instructs the server to transmit the data identified by the URL to the client.
    5. Should only retrieve data and should have no other effect: data should never be modified on the server as a result of a GET request.
    6. In this sense, a GET request is read-only, but of course, once the client receives the data, it is free to do any operation with it on its own side – for instance, format it for display.
    7. Get requests can be cached and bookmarked. It remains in the browser history and on proxies. GET requests should never be used when dealing with sensitive data.
    8. Get method when used in the method field, leaves entity body empty.
    HEAD
    1. Same as the GET method for a resource, but returns only the response headers without the response body.
    2. In other words, it only transfers status line and header section as a request.
    3. This is useful for retrieving meta-information written in response headers, without having to transport the entire content.
    4. HEAD method is much faster than GET method. In HEAD method much smaller amount of data is transferred. The HEAD method asks only for information about a document and not for the document itself.
    POST
    1. POST is often used when uploading a file or submitting a completed web form.
    2. Send data to the server for a new entity.
    3. The data is included in the body of the request.
    4. Submits data to be processed (e.g., from an HTML form) to the identified resource.
      POST requests should cause processing of the request body as a subordinate of the URL you are posting to.
    5. This may result in the creation of a new resource or the updates of existing resources or both.
    6. POST is used when the processing you wish to happen on the server should be repeated, if the POST request is repeated (that is, they are not idempotent).
    7. POST requests are never cached and bookmarked and do not remain in the browser history.
    PUT
    1. Similar to POST, but used to replace an existing entity.
    2. Replaces all current representations of the target resource with the request payload.
    3. A PUT request is used when you wish to create or update the resource identified by the URL.
    4. PUT requests contain the data to use in updating or creating the resource in the body.
      In cURL, you can add data to the request with the -d switch.

      1 curl -v -X PUT -d “some text”

    1 PUT /clients/robin

    might create a client, called Robin on the server.
    You will notice that REST is completely backend agnostic;
    there is nothing in the request that informs the server how the data should be created – just that it should.
    This allows you to easily swap the backend technology if the need should arise.

    DELETE Delete method sends a request to the server to perform delete operation on the specified resource.
    CONNECT
    1. Primarily used to establish a network connection to the server
      (usually via some proxy that can be requested to forward an HTTP request as TCP and maintain the connection).
    2. Converts the request connection to a transparent TCP/IP tunnel, usually to facilitate SSL-encrypted communication (HTTPS) through an unencrypted HTTP proxy.
    3. Once established, the response sends a 200 status code and a “Connection Established” message.
    OPTIONS
    1. For Web Services like RESTful APIs – lists down all the operations a web service supports
    2. Describes communication options for the target resource.
    3. Returns to the client information about the HTTP methods that the server supports for specified URL.
    4. It creates read-only requests to the server.
    5. Can be used to check the functionality of a web server (instead of a specific resource) by using a * wildcard in the URI.
    TRACE
    1. It performs message loop-back test along the path to the target resource.
    2. Provides a means to test what a machine along the network path receives when a request is made.
    3. As such, it simply returns what was sent.
      TRACE Echoes back the received request, so that a client can see what (if any) changes or additions have been made by intermediate servers.
    PATCH Similar to PUT, but used to update only certain fields within an existing entity.

    Is used to apply partial modification

    Methods Description
    Safe HEAD, GET, OPTIONS and TRACE
    1. They are intended only for information retrieval and should not change the state of the server.
    2. In other words, they should not have side effects, beyond relatively harmless effects such as logging, caching, the serving of banner advertisements or incrementing a web counter.
    3. Making arbitrary GET requests without regard to the context of the application’s state should therefore be considered safe.
    Unsafe POST, PUT, and DELETE.
    1. May result in a modification of the resources.
    2. Intended for actions which may cause side effects either on the server, or external side effects such as financial transactions or transmission of email.
    3. Such methods are therefore not usually used by conforming web robots or web crawlers, which tend to make requests without regard to context or consequences.

    1. Idempotent methods achieve the same result for multiple requests, no matter how many times the request is repeated.
      It does not matter if the request is called one or ten times, the result would be same.
    2. Methods PUT and DELETE are idempotent: multiple identical requests have the same effect as a single request.

      PUT and DELETE being considered idempotent might be surprising, though, it, in fact, is quite easy to explain:

      1. repeating a PUT method with exactly the same body should modify a resource in a way that it remains identical to the one described in the previous PUT request: nothing will change!
      2. Similarly, same resource can not be DELETEd twice.

      No matter how many times a PUT or DELETE request is repeated, the result is the same as if it had been done only once.

    3. Methods GET, HEAD, OPTIONS and TRACE, being prescribed as safe, should also be idempotent, as HTTP is a stateless protocol.
    4. The only non idempotent methods are PATCH and POST.

    1. The idea behind PUT request being idempotent is that if an update call on a resource fails, a client can make the same call again without causing any undesirable or inconsistent state.

      PUT request should always be preceded by a GET request to the resource, and should succeed if and if only resource hasn’t changed since.

    2. PATCH request is intended to update just selective fields, it is not expected to GET the resource representation. So multiple calls to PATCH request could end up in undesirable change in the resource state. Hence it is not IDEMPOTENT.

      It can be made idempotent though using etags or If Modified Since headers.

    3. PATCH changes the resources attributes. The change may require a concrete previous value of the attribute, which makes it NON idempotent.
      From Name=John to Name=Gargantua

      After repeated applying the name will be Gargantua and patch will fail, since it requires the name to be “John” before the change

      "from Name=John"
    4. JSON patch format: if the add operation is applied using the Patch method it is completely possible that the request is non-idempotent as it could add the new values to an existing resource if the same request is made multiple times.
      { "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] }
    5. So while most PATCH operations might be idempotent, there are some that aren’t.

    1. Both are methods used in HTTP requests.
    2. Generally it is said that GET is to download data and PUT is to upload data.
    3. But we can do both downloading as well as uploading either by GET/POST.
    Get Post
    It is cached. It cannot be cached.
    How parameters are sent It sends data using url in the browser.

    In GET, parameters are appended to the URL.

    It does not send data with the url.

    If we are using POST then we are sending parameters in the body section of a request.

    security If we are sending parameters in a GET request to the server, then those parameters will be visible in the URL.
    So there’s a lack of security while uploading to the server.
    If we send data after using encryption in the body of an http request, it’s quite a bit more secure.
    Data amount It can send limited amount of data to the server in a GET request
    because the URL has its max limit and we can not append a long data string to the URL.
    We can send data in bulk to the server.
    speed GET is faster in the case of just getting data using a static API call in cases where we don’t have to pass any parameters.

    1. GET appends data to the service URL.
      Its size shouldn’t exceed the maximum URL length.
      With GET method, you are limited to a maximum of 2,048 characters, minus the number of characters in the actual path.
    2. POST doesn’t have any such limit.
    3. So, theoretically, a user can pass unlimited data as the payload to POST method.
    4. However, if we consider a real use case, then sending POST with large payload will consume more bandwidth.
      It’ll take more time and present performance challenges to your server.
      Hence, a user should take action accordingly.

    1. The PUT or POST method should be used to create a resource.
    2. GET is only used to request data from a specified resource.

    PUT and POST operation are quite similar, except the terms of the result generated by them.

      • PUT operation is idempotent.
      • Responses to the PUT method are not cacheable.
        If a successful PUT request passes through a cache that has one or more stored responses for the effective request URI, those stored responses will be invalidated
      • The PUT method requests that the state of the target resource be created or replaced with the state defined by the representation enclosed in the request message payload.
      • Responses to POST operation not idempotent, and if you retry the request N times, you will end up having N resources with N different URIs created on server.

        However, If the result of processing a POST would be equivalent to a representation of an existing resource, an origin server MAY redirect the user agent to that resource by sending a 303 (See Other) response
        with the existing resource’s identifier in the Location field.

      • Responses to POST requests are only cacheable when they include explicit freshness information (see Section 4.2.1 of [RFC7234]). However, POST caching is not widely implemented.
      • The POST method requests that the target resource process the representation enclosed in the request according to the resource’s own specific semantics.

        For example, POST is used for the following functions (among others):

        o Providing a block of data, such as the fields entered into an HTML
        form, to a data-handling process;

        o Posting a message to a bulletin board, newsgroup, mailing list,
        blog, or similar group of articles;

        o Creating a new resource that has yet to be identified by the
        origin server; and

        o Appending data to a resource’s existing representation(s).

    Example:
    Let’s say we are designing a network application.
    Let’s list down a few URIs and their purpose to get to know when to use POST and when to use PUT operations.

    GET /device-management/devices : Get all devices
    POST /device-management/devices : Create a new device
    
    GET /device-management/devices/{id} : Get the device information identified by “id”
    PUT /device-management/devices/{id} : Update the device information identified by “id”
    DELETE /device-management/devices/{id} : Delete device by “id”
    

    As per IETF,
    “The fundamental difference between the POST and PUT methods is
    highlighted by the different intent for the enclosed representation.

    The target resource in a POST request is intended to handle the
    enclosed representation according to the resource’s own semantics,

    whereas the enclosed representation in a PUT request is defined as
    replacing the state of the target resource. Hence, the intent of PUT
    is idempotent and visible to intermediaries, even though the exact
    effect is only known by the origin server.

    Proper interpretation of a PUT request presumes that the user agent
    knows which target resource is desired.

    A service that selects a proper URI on behalf of the client, after receiving a state-changing
    request, SHOULD be implemented using the POST method rather than PUT.

    A PUT request applied to the target resource can have side effects on
    other resources. For example, an article might have a URI for
    identifying “the current version” (a resource) that is separate from
    the URIs identifying each particular version (different resources that at one point shared the same state as the current version resource).

    A successful PUT request on “the current version” URI might therefore create a new version resource in addition to changing the state of the target resource, and might also cause links to be
    added between the related resources.”

    Read more on IETF >>>

    Status/Response Codes in HTTP

    HTTP response codes standardize a way of informing the client about the result of its request.

    1. It is a Standard response code given by web server on Internet.
    2. It helps to identify the cause of a problem when web page or other resource does not load properly.
    3. In HTTP/1.0 and since, the first line of the HTTP response is called the status line and includes a numeric status code (such as “404”) and a textual reason phrase (such as “Not Found”).
    4. The way the user agent handles the response primarily depends on the code and secondarily on the response headers.
    5. Custom status codes can be used since, if the user agent encounters a code it does not recognize, it can use the first digit of the code to determine the general class of the response.

    There are two major groups of HTTP status code errors:

    1. 4xx Client Error
    2. 5xx Server Error

    Response Code Description
    200 Ok Http Response Code This response code indicates that the request was successful.
    201 Created Http Response Code 201 code indicates that the request was successful and a resource was created.

    It is used to confirm the success of a PUT or POST request.

    400 Bad Request Http Response Code The request was malformed.

    This happens especially with POST and PUT requests, when the data does not pass validation, or is in the wrong format.

    401 Unauthorized Http Response Code 401 error is generated in response to an unauthorized user request for a secure resource on the web server.

    It indicates that you need to perform authentication before accessing the resource.

    404 Not Found Http Response Code 404 code indicates that the requested resource is not available (could not be found) at the web server.

    This is generally returned to all requests which point to a URL with no corresponding resource.

    405 Method Not Allowed Http Response Code The HTTP method used is not supported (not allowed) for this resource.
    E.g., if a resource allows GET method, we cannot use POST to get this resource.
    409 Conflict Http Response Response Code This indicates a conflict.
    E.g., you are using a PUT request to create the same resource twice.
    500 Internal Server Error Web server displays 500 internal server error, when processing fails due to some unanticipated incident on the server side.

    When all else fails; generally, a 500 response is used when processing fails due to unanticipated circumstances on the server side, which causes the server to error out.

    Tools, Backups and Network Testing

    Network testing should be run

    1. ad-hoc after a configuration change to validate that everything went well
    2. as well as permanently, via active network monitoring, to detect network problems as soon as they happen.

    In the first case, here are some situations in which you want to validate your design and implementation assumptions after a configuration change:

    1. Circuits or site turn-up:
      once a new remote site or WAN link is installed, you can verify with a tool like iPerf that you get the bandwidth requested of your carrier and with ping to confirm that the circuit has no packet loss.
    2. Routing policy change:
      due to network complexity, the larger the network, the higher the risk that a routing policy change will have unexpected consequences on your routing table.

      By relying on distributed monitoring agents that run continuous ping and traceroute tests in a full-mesh fashion, you can validate in real-time that a routing policy change is modifying your routing table as expected.

    3. Firewall rules updates:
      it’s always good practice to verify that a new firewall ruleset is successfully implemented, whether it should be blocking, or allowing, certain traffic.

      To verify a successful update of a firewall, you can use a port scanner like nmap, or perform a TCP-based ping test from the unprotected to the protected network.

    4. Quality of Service (QoS):
      applying a QoS configuration to your network is not an easy task.
      There are so many dependencies and little things that could go wrong, so testing is extremely important to verify that, in the end, the network is classifying, marking, and queuing your traffic as designed.

    Nmap

    Nmap is the network security scanner of choice.
    It can give you useful information about what’s running on network hosts.
    It’s also so famous that it has been featured in many movies.

    With Nmap you can, for example, scan and identify open and filtered TCP/IP ports, check what operating system is running on a remote host, and do a ping sweeps on an IP subnet or range.

    List open ports on a host
    Knowing which TCP/IP ports of a host are listening for incoming connections is crucial, especially when you’re hardening a server or locking down network equipment. Nmap allows you to quickly verify that.

    Ping sweep on a IPv4 subnet

    Ping sweeps are great to create an inventory list of hosts in a network. Use it with caution and don’t simply scan the entire 10.0.0.0/8 subnet. Rather, go subnet per subnet (e.g. 10.1.1.0/24).

    Tcpdump

    Tpcdump is a great utility to troubleshoot network and application issues.

    Tcpdump is the tool that you want to use to analyze traffic sourced or destined to your own host or capture traffic between two or more endpoints (also called sniffing).

    To sniff traffic, you will need to connect the host running tcpdump to a SPAN port (also called port mirroring), to a hub (if you can still find one), or a network TAP.
    This will allow you to intercept and process all captured traffic with tcpdump.

    Iperf

    Use this tool to assess the bandwidth available between two computers. You can pick between TCP or UDP traffic and set the destination port, bandwidth rate (if UDP is selected), DSCP marking, and TCP window size. With iperf you can also generate multicast traffic and test your PIM infrastructure.

    Use Iperf to troubleshoot network slowness issues, verify whether the ISP would honor the DSCP marking, and measure the jitter value of UDP traffic.

    Hping3

    Hping3 is a utility command very similar to ping, with the difference that it can use TCP, UDP, and RAW-IP as transport protocols. Hping3 allows you to not only test if a specific TCP/IP port is open, but allows you to measure the round-trip time.

    Netcat

    Netcat (nc) is the network engineer’s Swiss Army knife. If you want to be the MacGyver of your network, you must know the basics of netcat. If you use it in client mode, it’s similar to telnet, and you can create a TCP connection to a specific port and send anything that you type. You can also use it to open a TCP/IP port and read from standard input. That makes it an easy way to transfer files between two computers. Another use case is testing whether your firewall is blocking certain traffic. For example, execute netcat in server mode on a host behind your firewall and then execute netcat in client mode from outside the firewall. If you can read on the server whatever you type on the client, then the firewall is not filtering the connection.

    SNMP based network management tools

    Most of these tools use the standard based SNMP protocol to interact with the network hardware and poll real-time status and usage of their resources, like processor utilization, memory consumption, and bytes transmitted and received on their interfaces. When a network node is unreachable or its resources are overloaded or not available, the network monitoring tool generates an alert to notify the administrator of the problem.

    In this class of tools there are many options available, and we can find open source solutions like Nagios and Zabbix as well as commercial ones like HP Network Node Manager (OpenView), SolarWinds, and IBM Tivoli.

    Pros

    1. Detect hardware failures and overload of system resources
    2. Provide bytes in/out network interfaces
    3. Always on monitoring (24×7)
    4. Fairly easy to setup

    Cons

    1. Lack of end-to-end visibility
    2. Not appropriate for troubleshooting network performance issues
    3. Hardly circumvent network complexity and virtualization
    4. Cannot detect software configuration errors (routing policies, ACL, …) that affect user traffic

    Flow based performance monitoring tools

    These tools can process and capture real user data (also called traffic flow) so you can obtain aggregate statistics about the protocols and users consuming a link capacity (top talkers) or inspect a specific sequence of packets to pinpoint performance issues between a client and a server. Traffic flows are captured by an inline device (network tap), a software agent, or network element that is switching the user traffic. The captured flow is then sent to a central collector for storage and processing. You can configure a mirror port on a switch to copy traffic flows for further analysis by a flow collector. Routers can also run NetFlow, SFlow and other type of protocols that generate statistics about user traffic.

    In this class, you can find source tools like NTOP and WireShark and commercial solutions like Riverbed and NetScout.

    Pros

    1. Accounting and statistics about traffic flows
    2. Protocols breakdown across network links
    3. Identify top talkers
    4. Deep packet inspection analysis

    Cons

    1. High disk space consumption
    2. Limited historical data
    3. In-line devices (taps) introduce another point of failure
    4. Taps are generally expensive so cannot be installed everywhere
    5. Mirror ports consume system resources and cannot capture all the flows traversing the node
    6. Require expertise and training
    7. Reactive troubleshooting

    This type of monitoring is accomplished by injecting real packets into the network to measure end-to-end reachability, round-trip-time, packet loss, bandwidth, link utilization and other network proprieties. Active monitoring is also used to test applications from the user perspective by executing real transactions and then measuring their performance like execution and response time. This technique enables you to test the end result of network and applications, without having to monitor individual components and then inferring their availability and performance. The feedback and detection of outages and performance degradation issues is much faster and more reliable.

    In this class you can find open source tools like SmokePing and Iperf as well as commercial solutions like NetBeez and AppNeta.

    Pros

    1. Detect performance degradation and trends
    2. Always on monitoring (24/7)
    3. Can hold large amount of historical data
    4. Does not require real user traffic to generate KPIs
    5. Test network infrastructure in the pre-deployment phase
    6. Validate configuration changes

    Cons

    1. In performing real transactions, these tools consumes network and/or application resources
    2. To be successfully implemented, several hardware or software agents must be deployed in the network

    User agent is a software that provides the service to the user to make the process of sending and receiving a message easier.

    User Agent Services:

    1. Composing messages
    2. Reading messages
    3. Replying to messages
    4. Forwarding messages
    5. Handling messages

    ipconfig is one of the primary network connection troubleshooting and information tools available for Windows Operating Systems. It allows the user to see what the current information is, force a release of those settings if set by DHCP, force a new request for a DHCP lease, and clear out the local DNS cache among other functions it is able to handle. ifconfig is a similar utility for Unix/Linux systems that while at first glance seems to be identical, it actually isn’t. It does allow for very quick (and thorough) access to network connection information, it does not allow for the DHCP functions that ipconfig does. These functions in fact are handled by a separate service/daemon called dhcpd.

    Shadow copies are a versioning system in place on Windows operating systems. This allows for users to go back to a previously available version of a file without the need for restoring the file from a standard backup- although the specific features of shadow copies vary from version to version of the OS. While it is not necessary to use a backup function in conjunction with Shadow Copies, it is recommended due to the additional stability and reliability it provides. Please note- Shadow Copies are not Delta Files. Delta files allow for easy comparison between versions of files, while Shadow Copies store entire previous versions of the files.

    Also known as the program that can give your admin nightmares, telnet is a very small and versatile utility that allows for connections on nearly any port.

    Telnet is an application that is used to connect to a remote host’s command line terminal interface.

    Network and system administrators use this application to configure and administer network devices such as servers, routers, switches, etc.

    This application is based on the connection-oriented Transmission Control Protocol (TCP).

    By default, a telnet server listens on port 23 for incoming connections from clients.

    All information exchanged in a telnet session between a client and server is unencrypted in cleartext (like ftp).
    For this reason, in the last years this application has been replaced by SSH, which provides the same type of service, but encrypted, as it’s based on the Secure Socket Layer (SSL) protocol.

    Although telnet has become obsolete for remote administration purposes
    this application is still widely used to verify connectivity to remote services that are based on TCP.

    In a client-server architecture, you can use telnet to make sure that no firewalls in between are blocking incoming connections to the server.

    Telnet Firewall check

    Telnet is used in cases where the user is trying to see if a program is listening on a particular port, but they want to keep a low profile.

    At a very basic level, there really isn’t one. As you progress up the chain however, you start to realize that there actually are a lot of differences in the power available to users (and admins) depending on how much you know about the different interfaces. Each of these utilities is a CLI- Command Line Interface- that allows for direct access to some of the most powerful utilities and settings in their respective operating systems.

    Command Prompt (cmd) is a Windows utility based very heavily on DOS commands, but has been updated over the years with different options such as long filename support.

    Bash (short for Bourne-Again Shell) on the other hand is the primary means of managing Unix/Linux operating systems and has a great deal more power than many of its GUI counterparts. Any Windows user that is used to cmd will recognize some of the commands due to the fact that DOS was heavily inspired by Unix and thus many commands have versions that exist in Bash. That being said, they may not be the best ones to use; for example while list contents (dir) exists in Bash, the recommended method would be to use list (ls) as it allows for much easier-to-understand formatting.

    Powershell, a newer Windows Utility, can be considered a hybrid of these two systems- allowing for the legacy tools of the command prompt with some of the much more powerful scripting functions of Bash.

    1. cURL is, more often than not, the HTTP client solution of choice for PHP developers.
    2. To experiment with the different request methods, you need a client, which allows you to specify which method to use.
    3. Unfortunately, HTML forms do not fit the bill, as they only allow you to make GET and POST requests.
    4. In real life, APIs are accessed programmatically through a separate client application, or through JavaScript in the browser.
    5. This is the reason why, in addition to the server, it is essential to have good HTTP client capabilities available in your programming language of choice.
    6. A very popular HTTP client library is, again, cURL.
      cURL includes both a standalone command line program, and a library that can be used by various programming languages.
    7. In particular, cURL is, more often than not, the HTTP client solution of choice for PHP developers.
    8. Other languages, such as Python, offer more native HTTP client libraries.

    What is cURL in HTTP?

    cURL is a command line tool.

    It is available on all major operating systems.

    External Media has been used for backups for a very long time, but has started to fall out of favor in the past few years due to its speed limitations.
    As capacities continue to climb higher and higher, the amount of time it takes to not only perform a backup but also a restore skyrockets.

    Tapes have been particularly hit hard in this regard, primarily because they were quite sluggish even before the jump to the terabyte era.

    Removable hard disks have been able to pick up on this trend however, as capacity and price have given them a solid lead in front of other options.

    But this takes us back to the question- why use EXTERNAL media?
    Internal media usually is able to connect faster, and is more reliable correct? Yes and no.

    While the estimated lifetime of storage devices has been steadily going up, there is always the chance for user error, data corruption, or hiccups on the hard disk.

    As a result, having regular backups to external media is still one of the best bang-for-buck methods available.

    Removable hard disks now have the capability to connect very rapidly, even without the use of a dedicated hot-swap drive bay.

    Through eSATA or USB3, these connections are nearly as fast as if they were plugged directly into the motherboard.

    Virtual Machines have only recently come into mainstream use, however they have been around under many different names for a long time. With the massive growth of hardware outstripping software requirements, it is now possible to have a server lying dormant 90% of the time while having other older systems at max capacity. Virtualizing those systems would allow the older operating systems to be copied completely and running alongside the server operating system- allowing the use of the newer more reliable hardware without losing any information on the legacy systems. On top of this, it allows for much easier backup solutions as everything is on a single server.

    RDP or Remote Desktop Protocol is the primary method by which Windows Systems can be remotely accessed for troubleshooting and is a software-driven method. KVM or Keyboard Video and Mouse on the other hand allows for the fast-switching between many different systems, but using the same keyboard, monitor and mouse for all. KVM is usually a hardware-driven system, with a junction box placed between the user and the systems in question- but there are some options that are enhanced by software. KVM also doesn’t require an active network connection, so it can be very useful for using the same setup on multiple networks without having cross-talk.

    A print server can refer to two different options- an actual server that shares out many different printers from a central administration point, or a small dedicated box that allows a legacy printer to connect to a network jack. A network attached printer on the other hand has a network card built into it, and thus has no need for the latter option. It can still benefit from the former however, as network attached printers are extremely useful in a corporate environment since they do not require the printer to be connected directly to a single user’s system.

    Devices

    Cables allow to form point-to-point network connections.
    These are networks where only a single device at each end of the link exists.

    Copper
    Cables
    Fiber
    Cables
    Made of Most common form of networking cables, made of multiple pairs of copper wires inside plastic insulator.

    The most common form of copper twisted-pair cables used in networking are Cat5(older), Cat5e and Cat6 cables.

    Contain individual optical fibers, which are tiny tubes made out of glass about the width of a human hair that can transport beams of light.
    How signal is sent The sending device communicates the binary data across copper wires by changing the voltage between two ranges.

    The system at the receiving end is able to interpret these voltage changes as binary ones and zeros which can then be translated into different forms of data.

    Use pulses of light to represent binary ones and zeros of the underlying data.
    Interference Differences in how the twisted pair cables are arranged inside can drastically alter how fast the data can be sent across them and how resistant these signals are to outside interference.

    Crosstalk – when an electrical pulse on one wire is accidentally detected on another wire.
    In this case the receiving end is not able to understand the data causing the network error.

    Higher level protocols have methods for detecting the missing data and asking for the data a second time.

    Used in environments when there is a lot of electromagnetic interference from outside sources, because this can impact data being sent across copper wires.
    Speed Transport data quicker than copper cables can.
    Distance Fiber can transport data over much longer distances than copper can without suffering the potential data loss.
    Fragility Fiber cables are much more expensive and fragile than copper.

    Multimode is cheaper but can’t transmit as far. Single Mode has a smaller core (the part that handles light) than Multimode, but is better at keeping the light intact. This allows it to travel greater distances and at higher bandwidths than Multimode. The problem is that the requirements for Single Mode are very specific and as a result it usually is more expensive than Multimode. Therefore for applications, you will usually see Multimode in the datacenter with Single Mode for long-haul connections.

    Hubs and switches are the primary devices used to connect computers on a single network, usually referred to as a LAN, or Local Area Network.

    Hub Network
    switch
    Layer Hub is a physical layer device that allows for connections from many computers at once. Network switch was originally known as a switching hub.
    It is similar to a hub because many devices can be connected to it.
    However while hub is layer 1 (physical) device, switch is layer 2 (data link) device.
    Collision domain All the devices connected to hub will end up talking to all other devices at the same time.
    It is up to each system connected to the hub to determine if the incoming data was meant for them or to ignore it if it wasn’t. This causes a lot of noise on the network and creates a collision domain.

    Collision domain – a network segment where only one device can communicate at a time.
    If multiple systems try to send data at the same time, the electrical pulses sent across the cable can interfere with each other.
    These system will have for a quiet period before they trying their data again.

    Being level 2 device, the switch can examine the contents of the Ethernet protocol data being sent around the network to determine which system the data is intended for and then only send that data to that one system.
    This reduces/eliminates the size of collision domains on the network
    Speed Collision domains slows down network communications and is the primary reason hubs are fairly rare. Reduction of collision domains leads fewer retransmissions and and a higher overall throughput.

    1. Sometimes a switch receives the packets faster than they can be accommodated in the shared link. These packets are stored in the memory for sometime.
    2. However, after sometime the buffer space will exhaust and some packets will start getting dropped off.
    3. The switch is called as congested in this state.

    Router is a network device that

    1. knows how to forward data between independent networks.
    2. connects two or more network segments.
    3. used to transfer information from the source to destination.

    Router operates at level 3 (Network).

    Routers are the global guides for getting traffic to the right places.

    Routers share data with each others via BGP (Border Gateway Protocol), which lets them learn about most optimal paths to forward traffic.

    Routers send the information in terms of data packets.
    When these data packets are forwarded from one router to another, router reads the network address in the packets and identifies the destination network.

    Router expects the IP data to determine where to send things.

    Routers store internal tables containing information about how to route traffic between a lot of different networks all over the world.

    Home network or a small office takes traffic originating from inside the home or office LAN and to forward it along to ISP (Internet Service Provider).

    Once traffic is at ISP a more sophisticated router takes over.

    These core routers form the backbone of Internet and are directly responsible of how we send and receive the data all over the Internet.
    A core router usually has many connections to many other routers.

    1. NIC stands for Network Interface Card.
    2. It is a peripheral card attached to a PC that helps to connect to a network.
    3. The MAC address of the card helps the network in identifying the PC.