Wisdom fish (AOOSANG)
Abstract: This document is generally introduced in the main components of DirectShow, as well as some of the basic concepts of DirectShow. Familiar with these basic knowledge will help DirectShow's application development or filter developers.
DirectShow is a development package provided by Microsoft's streaming of streaming on the Windows platform, which is posted with the DirectX development package. So what can DirectSHOW? It is also seen that DirectShow provides powerful support for the capture and playback of multimedia streams. Using DirectShow, we can easily capture data from the acquisition card that supports the WDM drive model, and performs corresponding post-processing or even stored in the file. It is widely supported in various media formats, including ASF, MPEG, AVI, DV, MP3, WAVE, etc., making the playback of multimedia data becomes light. In addition, DirectShow also integrates the technology of DirectX, such as DirectDraw, DirectSound, directly supports DVD playback, nonlinear editing of video, and exchange of data with digital cameras. What is more worth mentioning is that DirectShow provides an open development environment, we can customize your components according to your needs.
The relationship between the application and the DirectShow component and the hardware and hardware supported by DirectShow are shown in Figure 1.
Figure 1 DirectShow system block diagram
1 DirectShow's Filter
DirectShow is based on modular, and each function module takes a COM component, called Filter. DirectShow provides a range of standard modules available for application development, developers can also develop their own functionality for extension DirectShow applications. Below we use an example to explain how to take the Filter to play an AVI video file.
1 First read AVI data from one file to form a byte stream. (This work is completed by the source filter)
2 Check the head format of the AVI data stream, and then separate the video stream and the audio stream through the AVI split.
3 Decode the video stream, select different decoder filters depending on the compressed format.
4 Heavy video image via renderer filter.
5 Send the audio to the sound card to play, generally use the default DirectSound device filter. The flow is shown in the following figure.
From the above chart, each Filter is one other or two Filter. The connection points connected to the two FILTERs are also COM objects, we call PIN. Filter passes the data from a FILTER to another via the PIN, so that the data can be flowed in the linked list consisting of Filter. The arrows in the figure represent the direction of the data stream in the Filter linked list. In DirectShow, such a Filter Link on the top We are called Filter Graph.
Filter has three states, operation, stop, and pause. When a FILTER is running, it processes the media data stream. When it is stopped, the Filter is not processed, and the suspension state is often used to cure Data before the operation. Data Flow In The Filter Graph section describes these concepts in detail, you can refer to.
In addition to some special exceptions, the changes in the state of all Filter in Filter Graph are unified, that is, said that all Filter's state changes in Filte Graph are uniformly coordinated. In other words, we can also use Filter Graph to run, stop, suspend three states.
Filter is generally divided into several types below. (1) Source Filter: The source filter introduces data into the filter chart, the data source can be a file, network, camera, etc. Different source filters handle different types of data sources. (2) Transform filter: The operation of the transform filter is to obtain input stream, processing data, and generate an output stream. Transform filters The processing of data includes codec, format conversion, compression decompression, and the like. (3) Renderer filter: Submit filter in the final stage in the filter chart, and they receive data and submit data to peripherals. (4) Splitter filter: Split filter Segment the input stream into multiple outputs. For example, an AVI split filter divides an AVI format byte stream into a video stream and an audio stream. (5) MUX Filter: The mixed filter combines multiple input into a separate data stream. For example, the AVI mixed filter synthesizes the video stream and audio stream into an AVI format byte stream. These classifications of the filter are not absolute, such as an ASF reader filter (Asf Reader Filter)
A source filter is a split filter.
2 About FILTER GRAPH Manager
Filter Graph Manager is also a COM object that controls all of the filters in Filter Graph, which mainly has the following features:
1 Used to coordinate the status change between the filter, so that the changes in the state of all the filters in the Graph should be consistent.
2 Create a reference clock.
3 returning the Filter's message notification to the application
4 Provides a method for establishing a Filter Graph.
Here is just a simple description, you can refer to the document in detail.
Status change, the status change of the Filter in Graph should be consistent, so the application is sent directly to the file directly, but the command changed to the Filter Graph Manager, by manager to graph to graph Each Filter. Seeking is also working in the same way, first by the application to send the seek command to the Filter Graph Manager, then distribute it to each filter.
Reference clock, Filter in Graph uses the same clock, referred to as a reference clock, and the reference clock can ensure that all data stream synchronization, video or audio should be submitted as Presentation Time.Presentation Time It is determined relative to the reference clock. Filter Graph Manager should choose a reference clock, you can select the clock on the sound card, or select the system clock.
GRAPH event, the GRAPH manager uses event mechanism to notify the application that happened in the graph, which is similar to the WINDOWS message loop mechanism.
GRAPH build method, GRAPH manager provides an application with a method of adding a Filter to a GRAPH, a method of connecting to the Filter, disconnecting a Filter connection.
However, the GRAPH manager does not provide how to send data from a Filter to another Filter, this work is done independently by Filter internally by PIN.
3 media type
Because DirectShow is based on COM components, there is a way to describe the data format of each point of Filter Graph, for example, we also play the AVI file as an example, data into the graph in the form of the RIFF block, and then divided into a video And the audio stream, the video stream has a series of compressed video composition, after decompression, the video stream consists of a series of uncompressed bitmaps, and the audio stream also has the same step. Media Types: How DirectShow Repensents Formats
The media type is a very common method to describe how to describe digital media formats. When two Filter connections, they will adopt an agreement with a media type. The media type defines what kind of data that is in the source Filter will send downstream Filter, as well as the physical layout of the data. If the two Filter cannot support the same media type, then they can't connect.
For most applications, maybe you don't have to consider the media type, however, in some applications, you will be applied directly to the media type.
The media type is defined by the am_media_type structure to see the original definition.
Typedef struct _mediatype {
Guid Majortype;
Guid Subtype;
Bool bfixedsizesample;
Bool Btemporalcompression;
Ulong lsamplesize;
Guid FormatType;
Iunknown * punk;
Ulong cbformat;
[size_is (cbFormat)] byte * pbformat;
} AM_MEDIA_TYPE;
Major Type: is a GUID that defines the primary type of data, including, audio, video, unparsed byte stream, MIDI data, and more, and details can be referred to MSDN.
Subtype: Subtype, is also a GUID, which is used to further refine the data format, for example, in the video main type, including RGB-24, RGB-32, Uyvy, etc., some subtypes, including in the audio main type PCM AUDIO, MPEG-1 PayLoad, etc., subtypes provide more detailed information than the main type, but do not define all formats, for example, the subtype of the video does not define image sizes, rate. These are defined by the following fields.
Bfixedsizesamples indicates that the Sample size is fixed when this value is TRUE.
BtemPoralcompression When this value is True, it means that Sample uses a temporary compressed format, indicating that all is critical, if false, indicating that all is critical.
LsampleSize represents the size of the Sample. For compressed data, this value may be zero.
FormatType A GUID value used to indicate the format of the memory block. Including the following: format_none, format_dvinfo, format_mpegvideo, format_mpeg2video, format_videoinfo,
Format_videoInfo2, Format_WaveFormatex, Guid_null
Punk This parameter is not used
CBFormat memory size
PBFORMAT points to the pointer of the memory block,
Let's take a picture below to see how the Filter detects the media type. HRESULT CHECKMEDIATYPE (AM_MEDIA_TYPE * PMT)
{
IF (PMT == Null) Return E_POINTER;
// Check The Major Type. We're loops for video.
IF (PMT-> Majortype! = MediaType_Video)
{
Return VFW_E_INVALIDMEDITYPE;
}
// Check the subtype. We're looking for 24-bit rgb.
IF (PMT-> Subtype! = MediaSubType_RGB24)
{
Return VFW_E_INVALIDMEDITYPE;
}
// Check the format type and the size of the format block.
IF ((pmt-> formattype == format_videoinfo) &&&&
(PMT-> CBFORMAT> = Sizeof (VideoInfoHeader) &&
(PMT-> PBFORMAT! = null))
{
// Now it's safe to coercs the format block Pointer to The Format Block Pointer To
// Correct standure, as defined by the formattype guid.
VideoInfoHeader * PVIH = (VideoInfoHeader *) PMT-> PBFORMAT;
// Examine PVIH (Not Shown). If IT Looks OK, Return S_OK.
Return S_OK;
}
Return VFW_E_INVALIDMEDITYPE;
}
Let's take a brief introduction to several and media type related functions
The AM_MEDIA_TYPE structure contains a pointer to the data block, so when you use this structure, be careful in memory allocation to prevent memory leakage.
Assign function
1 am_media_type * WinAPI CreateMediaType
AM_MEDIA_TYPE Const * PSRC);
This function assigns a new AM_MEDIA_TYPE structure that contains data blocks in a particular format. Release the memory allocated by this function, you can call the deletemediatype. Function
2 stdapi createaudiomediatype
Const Waveformatex * PWFX,
AM_MEDIA_TYPE * PMT,
Bool bsetFormat;
This function uses a given WAVEFormatiex structure to initialize the media type. If the BsetFormat parameter is true, the function assigns a new memory. If the original PMT already contains memory, it is possible to have a memory leak. In order to avoid memory leaks, please call freeMediatype () before calling this function, then call FreeMediaType again after this function, release the Format Block.
3HRESULT WINAPI COPYMEDITYPE
AM_MEDIA_TYPE * PMTTARGET,
Const am_media_type * pmtsource);
This function copies a structure to another structure. This function also retesses the memory to the destination structure. If the PMTTARGET already contains a memory block, you have to spend a memory, so you want to call the FreeMediaType function before and after calling the function. Release function
4 void WinAPI deletemediatype (AM_MEDIA_TYPE * PMT);
Whether it is a CotaskMallAlloc function or a memory allocated with a createmediatype function, you can use this function to be released, if you don't connect the base class dynamic library, you can use the following code
void mydeletemediatype (am_media_type * pmt)
{
IF (PMT! = NULL)
{
MyFreeMediatype (* PMT); / / See the FreeMediatyPe function below
CotaskMemFree (PMT);
}
}
5 void WinAPI FreeMediatype (AM_MEDIA_TYPE & MT);
This function is used to release the memory of the data block. If you want to delete the AM_MEDIA_TYPE structure, you can use the deletemediatype function.
Void myfreemediatype (am_media_type & mt)
{
IF (mt.cbformat! = 0)
{
CotaskMemFree ((pvoid) mt.pbformat;
Mt.cbformat = 0;
Mt.pbformat = null;
}
IF (Mt.Punk! = NULL)
{
// Unecessary Because Punk Should Not Be Used, But Safest.
Mt.Punk-> Release ();
Mt.Punk = NULL;
}
}
4 media Samples and Allocators
Filters communicates data through the PIN connection, and the data stream is the input PIN that flows from a Filter's output PIN. The way to transfer data for PIN is to call the IMEMINPUTPIN :: Receive method on the input PIN.
For FILTER, there can be several ways to allocate the memory block used by media data, you can assign on the heap, can be on the surface of DirectDraw, or use GDI sharing memory, there are other methods, in DirectShow To make a memory allocation task is a memory distributor, a COM object, exposing an IMallocator interface.
When two PIN connections, there must be a PIN to provide an allocator. DirectShow defines a series of function calls to determine which PIN provides allocator, and the number of buffers.
Before the data stream begins, Allocator will create a memory of buffer. After starting the traffic data stream, the source filter will populate the data into an idle buffer in the memory pool, and then pass it to the following Filter. However, the source filter is not directly passed directly to the downstream file directly, but through a Media Samples COM object, this SAMPLE is created by allocator to manage memory buffers. Media Sample exposes the iMediaSample interface, a sample contains the following:
A pointer to the memory that is not sent.
A timestamp
Some sign
media type.
The timestamp indicates that the presentation time, rendere file is based on this time to schedule the render order. The logo is used to indicate whether the data is interrupted, and the media type provides a way to change the data format midway, however, general SAMPLE has no media type, indicating that their media type has not changed. When a Filter is using buffer, it keeps an Sample reference count, allocator is used by SAMPLE reference counts to determine if a buffer can be reused. This prevents the use of buffer, and when all Filter releases the reference to SAMPLE, SAMPLE returns to allocator's memory pool for reuse.
5 role of hardware equipment in graph
The following segment borrowed a document of Lu Qiming, this tag
2005-1-26 I think he is clear about the expression of hardware.
Everyone knows that in order to improve the stability of the system, the Windows operating system isolates hardware operations; applications generally cannot directly access hardware. DirectShow Filter works in user mode (User Mode, operating system privilege level is Ring 3), and hardware works in kernel mode (kernel mode, operating system privilege level is Ring 0), then how do they work together? DirectShow Solved Method is to package filter for these hardware; this Filter can work in user mode, appearance, control method, like normal filter, and the internal completion of the hardware driver inside the package filter driver. Such a design makes it a developer who writes a DirectShow application from a special process that needs to support hardware. DirectShow been integrated package Filter, including Audio Capture Filter (qcap.dll), VfW Capture Filter (qcap.dll, Filter Class Id of the CLSID_VfwCapture), TV Tuner Filter (KSTVTune.ax, Filter Class Id of the CLSID_CTVTunerFilter), Analog Video Crossbar Filter (KsxBar.ax), TV Audio Filter (Class ID is a CLSID_TVAUDIOFILTER), etc. In addition, DirectShow designs KsProxy Filter (KsProxy.ax,) for hardware with WDM drivers. Let's take a look at the structure.
From the above figure, we can see that KSProxy.ax, kstune.ax, ksxbar.ax These packaging FILTERs are in the same level with other ordinary DirectShow Filters, can work together; Filter in user mode controls hardware through stream class The driver minidriver (implemented by hardware vendors); Stream Class and MiniDriver provide the system underlayer level of service. It is worth noting that the stream class here is a driving model that is responsible for calling hardware MiniDriver; in addition, the Stream Class function is also coordinating between the MiniDriver, making some data can be directly transmitted from one hardware directly under Kernel Mode. To another hardware (or different functional modules on the same hardware), improve the working efficiency of the system. (More details about the underlying driver, please refer to Windows DDK.) Let's take a few more common hardware. VFW video capture card. This type of hardware is in a trend in the market; the newly produced video capture card generally uses WDM drive models. However, DirectShow is designed to keep backwards, but also provides a packaging filter to support this hardware. Like other hardware packaging filters, this packaging Filter creation is not using CocreateInstance like ordinary filter, and to enumerate through the system, then bindtoObject. Audio capture card (sound card). The acquisition function of the sound card is also implemented by packaging FILTER; and now most of the sound cards have a function of mixing. This Filter typically has several Input Pins, each represents an input, such as Line In, Microphone, CD, MIDI, etc. It is worth noting that these PIN representatives are the physical input terminals on the sound card, which will never connect to other filters in Filter Graph. The output function of the sound card can have two Filter supplies: DirectSound Renderer Filter and Audio Renderer (Waveout) FILTER. Note that these two FILTERs are not packaged filters in the above sense, they can interact with hardware because they use the API function: The former uses the DirectSound API, which uses the Waveout API. The difference between the two filters is that the latter output audio does not support mixing. (By the way, Video Renderer Filter can access the graphics card, because of the use of GDI, DirectDraw or Direct3D API.) If you have a sound card on your machine, you can see this sound card in the audio capture source in Audio Capture Sources. Filter. WDM driven hardware (including video capture card, hardware decompression card, etc.). This type of hardware uses KSProxy.ax this packaging filter. KsProxy.ax achieved a lot of features, so there is a reputation of the "Swiss Army"; it is also called "chameleon Filter" because the Filter defines a unified interface, and the implementation of the interface varies depending on the specific hardware driver. In Filter Graph, the name of KsProxy Filter displays hardware's Friendly Name (generally defined in the driver .inf file).
We can find WDM hardware installed in the native system in the directory at the beginning of WDM stream. Because KSProxy.ax can represent audio and video devices of various WDMs, this packaging Filter workflow is a bit complicated. This Filter does not know which type of device to represent, which must first access the attribute set of the driver, and then dynamically configure the interface that the FILTER should implement. When the interface method on the KSProxy Filter is called by the application or other Filter, it passes the calling method and the parameter to the driver, and the driver finally completes the specified function. In addition, WDM hardware also supports kernel streaming, ie, data transfer in core mode, without the conversion of user mode. Because the mutual conversion between kernel mode and user mode requires a lot of calculation. If the kernel stream is used, it can not only avoid a lot of calculations, but also avoid copying process between kernel data and host memory. In this case, in the filter graph of the user mode, even if the PIN is connected, there is no actual data flow. Typical situations, such as video capture cards with Video Port PIN, the image displayed when Preview is in the memory of the graphics card in the kernel mode. So, you also want to intercept the data stream behind VP Pin. Telling here, I think everyone should have a general understanding of the hardware support problem of DirectShow. For application developers, this area does not have to be treated too through, and just as background knowledge is better. In fact, a lot of cumbersome work DirectShow has helped us do well. These documents are the learning notes I organized during the DirectShow development process, and the content mainly involves the development of audio and video, I hope to help you.
