1.前言
本文介绍了webrtc中的fec相关封装原理, 协议，分析其在webrtc中的应用过程和使用策略。
2.正文
2.1 red
为什么做red封装呢？Ulpfec编码后的内容会做Red封装后再放入RtpPacket，可fec在RFC5109已经定义好自己的传输格式，而且sdp协商过程中也有Ulpfec的PT为a=rtpmap:125 ulpfec/90000, 可以通过RtpHeader的PT进行区分出Ulpfec包，看起来是多此一举. 其实不然，fec过程中要传输的包有两种，一种是冗余包，一种是原始包，将他们统一的放在red中，表示这些包属于一个fec流，然后red提供的pt可以分辨出那些是fec包，哪些是原始数据包
2.1.1 red封装格式
Red的封装如下,由block header + data block组成，payload会被放在data block中
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F| block PT | timestamp offset | block length | : block header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| block | : data block
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
介绍一下各个标志位
F: 为1时标识后面是否有别的block,如果有则会出现多个这样的header line
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F| block PT | timestamp offset | block length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F| block PT | timestamp offset | block length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.
.
.
为0时，标识后面不会再有这个header line，同时最后block的timestamp和block
length能够通过rtp推断出，所以会省略:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| block PT=7 | timestamp offset | block length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| block PT=5 | |
+-+-+-+-+-+-+-+-+
block PT: block内容中的rtp payload type
block length: block的大小
timestamp offset: block 的timestam 相对于rtp header timestamp 的offset
一个完整的red如下所示：
// 0 1 2 3
// 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// |V=2|P|X| CC=0 |M| PT | sequence number of primary |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// | timestamp of primary encoding |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// | synchronization source (SSRC) identifier |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// |1| block PT=7 | timestamp offset | block length |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// |0| block PT=5 | |
// +-+-+-+-+-+-+-+-+ +
// | |
// + LPC encoded redundant data (PT=7) +
// | (14 bytes) |
// + +---------------+
// | | |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
// | |
// + +
// | |
// + +
// | |
// + +
// | DVI4 encoded primary data (PT=5) |
// + (84 bytes, not to scale) +
// / /
// + +
// | |
// + +
// | |
// + +---------------+
// | |
由于webrtc只用到了单个block的red封装，就是在payload前加多1Byte，把payloadtype放进去
2.1.2 red封装流程
RTPSenderVideo::SendVideo()是对packet进行RTP打包的地方，会检查是否开启了red封装，会将打好的包重新设置成red_packet
bool RTPSenderVideo::SendVideo(
int payload_type,
absl::optional<VideoCodecType> codec_type,
uint32_t rtp_timestamp,
int64_t capture_time_ms,
rtc::ArrayView<const uint8_t> payload,
RTPVideoHeader video_header,
absl::optional<int64_t> expected_retransmission_time_ms,
absl::optional<int64_t> estimated_capture_clock_offset_ms) {
if (red_enabled()) {
// 将packet替换成red_packet，将media payload放进red_packet中
// TODO(sprang): Consider packetizing directly into packets with the RED
// header already in place, to avoid this copy.
std::unique_ptr<RtpPacketToSend> red_packet(new RtpPacketToSend(*packet));
BuildRedPayload(*packet, red_packet.get()); // 生成red包
red_packet->SetPayloadType(*red_payload_type_);// 设定payloadType为Red
red_packet->set_is_red(true);// 标识此包为一个red包
// Append |red_packet| instead of |packet| to output.
red_packet->set_packet_type(RtpPacketMediaType::kVideo);
red_packet->set_allow_retransmission(packet->allow_retransmission());
rtp_packets.emplace_back(std::move(red_packet));
} else {
packet->set_packet_type(RtpPacketMediaType::kVideo);
rtp_packets.emplace_back(std::move(packet));
}
....
}
这里首先会受到enable red的影响，去标识当前包要做red封包(rfc2189), 其中：
-
red_packet本身的类型和普通包的类型都是RtpPacketToSend
-
调用
BuildRedPayload()去做Red封装, 首先在此介绍一下Red封装void BuildRedPayload(const RtpPacketToSend& media_packet, RtpPacketToSend* red_packet) { // 新增1Byte，用于放Red Header uint8_t* red_payload = red_packet->AllocatePayload( kRedForFecHeaderLength + media_packet.payload_size()); RTC_DCHECK(red_payload); red_payload[0] = media_packet.PayloadType(); //填入payload type // 拷贝payload 到block上 auto media_payload = media_packet.payload(); memcpy(&red_payload[kRedForFecHeaderLength], media_payload.data(), media_payload.size()); } -
通过red_packet->set_is_red(true)表示了这是一个red封装的包
-
通过red_packet->SetPayloadType(*red_payload_type_) 设置了当前包的payloadType为red_payload_type，表征了payload为red，这个值一般是127, 这个config来自于struct UlpfecConfig
struct UlpfecConfig {
UlpfecConfig()
: ulpfec_payload_type(-1),
red_payload_type(-1),
red_rtx_payload_type(-1) {}
std::string ToString() const;
bool operator==(const UlpfecConfig& other) const;
// Payload type used for ULPFEC packets.
int ulpfec_payload_type;
// Payload type used for RED packets.
int red_payload_type;
// RTX payload type for RED payload.
int red_rtx_payload_type;
};
2.2 fec
2.2.1 ulpfec原理
fec的原理很简单大家都知道，就是异或，比如说传输两个包 P1 和 P2, 通过 异或P1，P2得到FEC包 F1, 那么传输过程中能收到任意两个，都可以通过异或恢复出原来的 P1 和 P2。
两个包的问题好考虑，但是多个包的时候会引入第二个问题，为了带宽控制fec包的数量时候，应该选择哪一部分包作为一个集合去异或出一个fec包呢? 比如有10个数据包，当前一段传输一段时间内丢包率为30%，此时可以引入一个简单的处理策略: 将fec包的数量控制为丢包率也就是3个，但这3个FEC包由哪一块做异或呢？ 如下所示，有两种分配方式，一种是以相邻的包为一组做fec，这时候每3个包可能丢一个，不影响恢复，这种模型能较好的对抗随机丢包的网络，第二种则是取等间隔的包做一组，突发性丢包会导致连续的一块包丢失，这种模型对突发丢包的网络比较好
2.2.2 ulpfec封装格式
详见RFC5109, 在Rtp传输时上，跟在RTPheader后，可以理解作为payload吧
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header (12 octets or more) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Header (10 octets) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Level 0 Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Level 0 Payload |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Level 1 Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Level 1 Payload |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Cont. |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
其中 FEC Header如下所示:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|E|L|P|X| CC |M| PT recovery | SN base |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TS recovery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| length recovery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
E: 标识是否启用扩展头，当前值为0，保留位，
L: 使用的mask表的长度，0-->16bits, 1-->48bits
P, X, CC, M, PT recovery: 进行异或的Rtp包头上的P, X, CC, M, PT的异或结果
SN base: 进行异或的Rtp包的最小sequence号
Ts recovery: 进行异或的Rtp包的Timestamp的异或结果
length: 进行异或的Rtp包的payload长度的异或结果
ulpfec中可以将不同的数据使用不同的level级别去做保护, 如下:
Packet A #####################
: :
Packet B ############### :
: :
ULP FEC Packet #1 @@@@@@@@ :
: :
Packet C ########### :
: :
Packet D ###################################
: :
ULP FEC Packet #2 @@@@@@@@@@@@@@@@@
: : :
:<-L0->:<--L1-->:
Payload packet # | ULP FEC packet that protects at level
| L0 L1
---------------------+---------------------------------------
A | #1 #2
B | #1 #2
C | #2 #2
D | #2 #2
对A和B的前半段数据使用#1包去保护，对C和D的前半段数据用#2包去保护，这前半段数据都处于L0级别
对于A,B,C,D的后半段数据使用#2包保护，这些保护的数据被设在L1级别
这里认为A,B,C,D的后半段的数据没那么重要，用4:1的比例做冗余就够了，而前半段的数据很重要，所以需要2:1的比例做冗余(一直觉得RFC上这图画的是不是有问题，L1的长度这么短，后面的数据是不要了吗，嘻)；
因为一个FEC包中可能有不同level级别的数据，为此在Fec header中引入了 FEC Level 0 Header 和 FEC Level 1 Header，结构如下所示，主要是标记了保护的数据长度和参与fec包的掩码表
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Protection Length | mask |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| mask cont. (present only when L = 1) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Protection Length： L0包保护的长度
mask: 标记packet是否参与异或的掩码表，每个bit标记着以fec header的SN base作为偏移起始的rtp Sequence对应的包是否参与当前fec block的编码
mask cont：当L被设置成1的时候 mask cont会出现和mask连起来形成48bit的掩码表
但这里需要明确一下，webrtc中没有用到L1这个级别，因为packet中的信息都是帧的一部分优先级都是一样的，但并不是说L1这个东西不能用，可以把上面的粒度替换一下，packet换成Frame，后面的数据换成包也可以尝试一番。
2.2.3 ulpfec编码过程
2.2.3.1 解red封装
webrtc做fec冗余的地方在随后发送链路中的RtpSenderEgress::SendPacket()
void RtpSenderEgress::SendPacket(RtpPacketToSend* packet,
const PacedPacketInfo& pacing_info) {
....
//fec处理
if (fec_generator_ && packet->fec_protect_packet()) {
// This packet should be protected by FEC, add it to packet generator.
RTC_DCHECK(fec_generator_);
RTC_DCHECK(packet->packet_type() == RtpPacketMediaType::kVideo);
absl::optional<std::pair<FecProtectionParams, FecProtectionParams>>
new_fec_params;
{
MutexLock lock(&lock_);
new_fec_params.swap(pending_fec_params_);
}
// fec_rate和fec_max_frame 可能被更新了
if (new_fec_params) {
fec_generator_->SetProtectionParameters(new_fec_params->first,
new_fec_params->second);
}
if (packet->is_red()) {
// 对packet做了red封装(rfc2198)，需要解red封装得到原始rtp包
// 很奇怪，会进行red封包的应该只有fec,普通包也会进行fec封装？
// 复制整个包
RtpPacketToSend unpacked_packet(*packet);
const rtc::CopyOnWriteBuffer buffer = packet->Buffer();
// Grab media payload type from RED header.
const size_t headers_size = packet->headers_size();
unpacked_packet.SetPayloadType(buffer[headers_size]);
// 对copyonwirte的payload进行拷贝
// Copy the media payload into the unpacked buffer.
uint8_t* payload_buffer =
unpacked_packet.SetPayloadSize(packet->payload_size() - 1);
std::copy(&packet->payload()[0] + 1,
&packet->payload()[0] + packet->payload_size(), payload_buffer);
// 对包做fec
fec_generator_->AddPacketAndGenerateFec(unpacked_packet);
} else {
// If not RED encapsulated - we can just insert packet directly.
fec_generator_->AddPacketAndGenerateFec(*packet);
}
}
.....
}
主要做了两件事:
- 通过packet->fec_protect_packet()检查packet是否启用了fec，是则使用fec_generator_->AddPacketAndGenerateFec()将包加入到fec的队列中准备做fec
- 如果packet做了red封装，需要对payload解red封装，也就是那个1 Byte的red header给去掉
2.2.3.2 更新fec params
此处用的是ulpfec，接下来的函数是UlpfecGenerator::AddPacketAndGenerateFec()
void UlpfecGenerator::AddPacketAndGenerateFec(const RtpPacketToSend& packet) {
RTC_DCHECK_RUNS_SERIALIZED(&race_checker_);
RTC_DCHECK(generated_fec_packets_.empty());
{
MutexLock lock(&mutex_);
// 更新fec params
if (pending_params_) {
current_params_ = *pending_params_;
pending_params_.reset();
// 重设的阈值高于kHighProtectionThreshold, 更改最少需要4个包去做fec encode
if (CurrentParams().fec_rate > kHighProtectionThreshold) {
min_num_media_packets_ = kMinMediaPackets;
} else {
min_num_media_packets_ = 1;
}
}
}
if (packet.is_key_frame()) {
media_contains_keyframe_ = true;
}
const bool complete_frame = packet.Marker();
if (media_packets_.size() < kUlpfecMaxMediaPackets) {
// 构造fec packet放入等待队列中
// Our packet masks can only protect up to |kUlpfecMaxMediaPackets| packets.
auto fec_packet = std::make_unique<ForwardErrorCorrection::Packet>();
fec_packet->data = packet.Buffer();
media_packets_.push_back(std::move(fec_packet));
// Keep a copy of the last RTP packet, so we can copy the RTP header
// from it when creating newly generated ULPFEC+RED packets.
RTC_DCHECK_GE(packet.headers_size(), kRtpHeaderSize);
last_media_packet_ = packet;
}
if (complete_frame) {
++num_protected_frames_;
}
auto params = CurrentParams();
// Produce FEC over at most |params_.max_fec_frames| frames, or as soon as:
// (1) the excess overhead (actual overhead - requested/target overhead) is
// less than |kMaxExcessOverhead|, and
// (2) at least |min_num_media_packets_| media packets is reached.
if (complete_frame &&
(num_protected_frames_ >= params.max_fec_frames ||
(ExcessOverheadBelowMax() && MinimumMediaPacketsReached()))) {
// We are not using Unequal Protection feature of the parity erasure code.
constexpr int kNumImportantPackets = 0;
constexpr bool kUseUnequalProtection = false;
// fec编码
fec_->EncodeFec(media_packets_, params.fec_rate, kNumImportantPackets,
kUseUnequalProtection, params.fec_mask_type,
&generated_fec_packets_);
if (generated_fec_packets_.empty()) {
ResetState();
}
}
}
主要做了:
-
更新ulpfec的params，结构如下，对于关键帧和P帧各有一套独立的控制参数，控制参数主要有
fec_rate: 为(fec包数)/(媒体包数) * 256，调整策略见 2.3
max_fec_frames: fec编码是针对收集到的一段packet进行fec编码的，此参数规定到了这个数量时一定要fec编码了
fec_mask_type：webrtc根据丢包模型准备了两个fec的掩码表，随机丢包掩码表和突发丢包掩码表
struct Params { Params(); Params(FecProtectionParams delta_params, FecProtectionParams keyframe_params); FecProtectionParams delta_params; // p帧 FecProtectionParams keyframe_params;// 关键帧 }; struct FecProtectionParams { int fec_rate = 0; // fec_packet_num / media_packet_num * 256 int max_fec_frames = 0; // 经过max_fec_frames一定要生成fec包 FecMaskType fec_mask_type = FecMaskType::kFecMaskRandom;// 选择使用给的fec掩码表 }; -
当fec_rate大于kHighProtectionThreshold(80)时，重设了最小FEC编码packet数为4，不太清楚这样改动的意义，是怕太频繁重复encode吗
-
构建Fec packet，将payload拷贝进去
-
检查是否立即进行FEC encode，满足的条件有一下两种，满足任意一个就调用fec_->EncodeFec()进行编码
1.完整帧 + 收集的帧数量超过设定值
2.完整帧 + 达到设定最小收集包数 + 当前做FEC所得到的rate和预设的fec_rate相减小于一个阈值
关于加黑的条件也就是ExcessOverheadBelowMax()，有一些细节, 在实际计算要用到的fec包的数量时，在fec_rate的基础上还加了个0.5的上取整，这是导致实际和预设出现差值的原因
// 通过fec_rate计算实际需要用到的fec包数 int ForwardErrorCorrection::NumFecPackets(int num_media_packets, int protection_factor) { // Result in Q0 with an unsigned round. // fec = media * fec_rate / 256 + 0.5 此处做了个上取整 int num_fec_packets = (num_media_packets * protection_factor + (1 << 7)) >> 8; // Generate at least one FEC packet if we need protection. if (protection_factor > 0 && num_fec_packets == 0) { num_fec_packets = 1; } RTC_DCHECK_LE(num_fec_packets, num_media_packets); return num_fec_packets; } int UlpfecGenerator::Overhead() const { RTC_DCHECK_RUNS_SERIALIZED(&race_checker_); RTC_DCHECK(!media_packets_.empty()); int num_fec_packets = fec_->NumFecPackets(media_packets_.size(), CurrentParams().fec_rate); // overheade的计算，用实际需要使用的 fec包数 * 256 / media包数 return (num_fec_packets << 8) / media_packets_.size(); } bool UlpfecGenerator::ExcessOverheadBelowMax() const { RTC_DCHECK_RUNS_SERIALIZED(&race_checker_); // 实际 - 预设 < 50 return ((Overhead() - CurrentParams().fec_rate) < kMaxExcessOverhead); }同时，注意到注释中说没有使用Unequal Protection feature of the parity erasure code.没有启动L1级别做FEC；
2.2.3.3 开始Encodefec
接下来到了实际的fec编码过程ForwardErrorCorrection::EncodeFec()
int ForwardErrorCorrection::EncodeFec(const PacketList& media_packets,
uint8_t protection_factor,
int num_important_packets,
bool use_unequal_protection,
FecMaskType fec_mask_type,
std::list<Packet*>* fec_packets) {
const size_t num_media_packets = media_packets.size();
// Sanity check arguments.
RTC_DCHECK_GT(num_media_packets, 0);
RTC_DCHECK_GE(num_important_packets, 0);
RTC_DCHECK_LE(num_important_packets, num_media_packets);
RTC_DCHECK(fec_packets->empty());
const size_t max_media_packets = fec_header_writer_->MaxMediaPackets();
if (num_media_packets > max_media_packets) {
RTC_LOG(LS_WARNING) << "Can't protect " << num_media_packets
<< " media packets per frame. Max is "
<< max_media_packets << ".";
return -1;
}
// Error check the media packets.
for (const auto& media_packet : media_packets) {
RTC_DCHECK(media_packet);
if (media_packet->data.size() < kRtpHeaderSize) {
RTC_LOG(LS_WARNING) << "Media packet " << media_packet->data.size()
<< " bytes "
"is smaller than RTP header.";
return -1;
}
// Ensure the FEC packets will fit in a typical MTU.
if (media_packet->data.size() + MaxPacketOverhead() + kTransportOverhead >
IP_PACKET_SIZE) {
RTC_LOG(LS_WARNING) << "Media packet " << media_packet->data.size()
<< " bytes "
"with overhead is larger than "
<< IP_PACKET_SIZE << " bytes.";
}
}
// Prepare generated FEC packets.
int num_fec_packets = NumFecPackets(num_media_packets, protection_factor);
if (num_fec_packets == 0) {
return 0;
}
for (int i = 0; i < num_fec_packets; ++i) {
generated_fec_packets_[i].data.EnsureCapacity(IP_PACKET_SIZE);
memset(generated_fec_packets_[i].data.MutableData(), 0, IP_PACKET_SIZE);
// Use this as a marker for untouched packets.
generated_fec_packets_[i].data.SetSize(0);
fec_packets->push_back(&generated_fec_packets_[i]);
}
// 根据丢包模型选择mask表，生成mask
internal::PacketMaskTable mask_table(fec_mask_type, num_media_packets);
packet_mask_size_ = internal::PacketMaskSize(num_media_packets);
memset(packet_masks_, 0, num_fec_packets * packet_mask_size_);
// 生成mask表
internal::GeneratePacketMasks(num_media_packets, num_fec_packets,
num_important_packets, use_unequal_protection,
&mask_table, packet_masks_);
// Adapt packet masks to missing media packets.
int num_mask_bits = InsertZerosInPacketMasks(media_packets, num_fec_packets);
if (num_mask_bits < 0) {
RTC_LOG(LS_INFO) << "Due to sequence number gaps, cannot protect media "
"packets with a single block of FEC packets.";
fec_packets->clear();
return -1;
}
packet_mask_size_ = internal::PacketMaskSize(num_mask_bits);
// 开始FEC
// Write FEC packets to |generated_fec_packets_|.
GenerateFecPayloads(media_packets, num_fec_packets);
// TODO(brandtr): Generalize this when multistream protection support is
// added.
const uint32_t media_ssrc = ParseSsrc(media_packets.front()->data.data());
const uint16_t seq_num_base =
ParseSequenceNumber(media_packets.front()->data.data());
FinalizeFecHeaders(num_fec_packets, media_ssrc, seq_num_base);
return 0;
}
ForwardErrorCorrection::EncodeFec()主要:
- 如2.2.1中所介绍的，随机丢包和突发丢包下fec组包模式不同，所以webrtc准备了两张mask表 kFecMaskRandom(随机丢包), kFecMaskBursty(突发丢包)去生成mask, 调用internal::GeneratePacketMasks()去生成mask
- 根据mask和packet, 调用GenerateFecPayloads()生成fec包
- 调用FinalizeFecHeaders()填入ssrc
2.2.3.4 生成mask
以下为调用要生成mask的internal::GeneratePacketMasks()中
void GeneratePacketMasks(int num_media_packets,
int num_fec_packets,
int num_imp_packets,
bool use_unequal_protection,
PacketMaskTable* mask_table,
uint8_t* packet_mask) {
RTC_DCHECK_GT(num_media_packets, 0);
RTC_DCHECK_GT(num_fec_packets, 0);
RTC_DCHECK_LE(num_fec_packets, num_media_packets);
RTC_DCHECK_LE(num_imp_packets, num_media_packets);
RTC_DCHECK_GE(num_imp_packets, 0);
const int num_mask_bytes = PacketMaskSize(num_media_packets);
// Equal-protection for these cases.
if (!use_unequal_protection || num_imp_packets == 0) {
// Retrieve corresponding mask table directly:for equal-protection case.
// Mask = (k,n-k), with protection factor = (n-k)/k,
// where k = num_media_packets, n=total#packets, (n-k)=num_fec_packets.
// 调用PacketMaskTable::LookUp
rtc::ArrayView<const uint8_t> mask =
mask_table->LookUp(num_media_packets, num_fec_packets);
memcpy(packet_mask, &mask[0], mask.size());
} else { // UEP case
UnequalProtectionMask(num_media_packets, num_fec_packets, num_imp_packets,
num_mask_bytes, packet_mask, mask_table);
} // End of UEP modification
} // End of GetPacketMasks
由于没有使用unequal protection，所以直接进入PacketMaskTable::LookUp()
PacketMaskTable::PacketMaskTable(FecMaskType fec_mask_type,
int num_media_packets)
: table_(PickTable(fec_mask_type, num_media_packets)) {}
PacketMaskTable::~PacketMaskTable() = default;
rtc::ArrayView<const uint8_t> PacketMaskTable::LookUp(int num_media_packets,
int num_fec_packets) {
RTC_DCHECK_GT(num_media_packets, 0);
RTC_DCHECK_GT(num_fec_packets, 0);
RTC_DCHECK_LE(num_media_packets, kUlpfecMaxMediaPackets);
RTC_DCHECK_LE(num_fec_packets, num_media_packets);
if (num_media_packets <= 12) {
// 小于12直接查表
return LookUpInFecTable(table_, num_media_packets - 1, num_fec_packets - 1);
}
int mask_length =
static_cast<int>(PacketMaskSize(static_cast<size_t>(num_media_packets)));
// Generate FEC code mask for {num_media_packets(M), num_fec_packets(N)} (use
// N FEC packets to protect M media packets) In the mask, each FEC packet
// occupies one row, each bit / coloumn represent one media packet. E.g. Row
// A, Col/Bit B is set to 1, means FEC packet A will have protection for media
// packet B.
// 大于12，使用interleaved，也就是间隔包为一组(X % N)
// Loop through each fec packet.
for (int row = 0; row < num_fec_packets; row++) {
// Loop through each fec code in a row, one code has 8 bits.
// Bit X will be set to 1 if media packet X shall be protected by current
// FEC packet. In this implementation, the protection is interleaved, thus
// media packet X will be protected by FEC packet (X % N)
for (int col = 0; col < mask_length; col++) {
fec_packet_mask_[row * mask_length + col] =
((col * 8) % num_fec_packets == row && (col * 8) < num_media_packets
? 0x80
: 0x00) |
((col * 8 + 1) % num_fec_packets == row &&
(col * 8 + 1) < num_media_packets
? 0x40
: 0x00) |
((col * 8 + 2) % num_fec_packets == row &&
(col * 8 + 2) < num_media_packets
? 0x20
: 0x00) |
((col * 8 + 3) % num_fec_packets == row &&
(col * 8 + 3) < num_media_packets
? 0x10
: 0x00) |
((col * 8 + 4) % num_fec_packets == row &&
(col * 8 + 4) < num_media_packets
? 0x08
: 0x00) |
((col * 8 + 5) % num_fec_packets == row &&
(col * 8 + 5) < num_media_packets
? 0x04
: 0x00) |
((col * 8 + 6) % num_fec_packets == row &&
(col * 8 + 6) < num_media_packets
? 0x02
: 0x00) |
((col * 8 + 7) % num_fec_packets == row &&
(col * 8 + 7) < num_media_packets
? 0x01
: 0x00);
}
}
return {&fec_packet_mask_[0],
static_cast<size_t>(num_fec_packets * mask_length)};
}
PacketMaskTable::LookUp()主要:
-
在media packet <= 12时，直接从系统预设的两张表查mask
-
在media packet > 12时，采用间隔的方式进行分组fec
其中查表的过程有一些细节,以kPacketMaskRandomTbl表为例，其定义如下：
const uint8_t kPacketMaskRandomTbl[] = {
12,
1, kMaskRandom1_1,
2, kMaskRandom2_1, kMaskRandom2_2
3, kMaskRandom3_1, kMaskRandom3_2, kMaskRandom3_3
4, kMaskRandom4_1, kMaskRandom4_2, kMaskRandom4_3, kMaskRandom4_4
kPacketMaskRandom5,
kPacketMaskRandom6,
kPacketMaskRandom7,
kPacketMaskRandom8,
kPacketMaskRandom9,
kPacketMaskRandom10,
kPacketMaskRandom11,
kPacketMaskRandom12,
};
以上手动对其中一部分做了宏定义展开，最高12表示该表支持的原始包的长度为12，然后接下来的每一行，是每个原始包的对应数量下不同数量fec包的mask表，比如说kMaskRandom4_3是原始包数量为4fec包数量为3情况下的一个大小为(4x3)掩码表的起始，理解这个表后，下面的找表函数就好理解了
rtc::ArrayView<const uint8_t> LookUpInFecTable(const uint8_t* table,
int media_packet_index,
int fec_index) {
RTC_DCHECK_LT(media_packet_index, table[0]);
// Skip over the table size.
const uint8_t* entry = &table[1];
uint8_t entry_size_increment = 2; // 0-16 are 2 byte wide, then changes to 6.
// Hop over un-interesting array entries.
// 跳过行
for (int i = 0; i < media_packet_index; ++i) {
if (i == 16)
entry_size_increment = 6;
uint8_t count = entry[0];
++entry; // skip over the count.
for (int j = 0; j < count; ++j) {
entry += entry_size_increment * (j + 1); // skip over the data.
}
}
if (media_packet_index == 16)
entry_size_increment = 6;
RTC_DCHECK_LT(fec_index, entry[0]);
++entry; // Skip over the size.
// Find the appropriate data in the second dimension.
// 跳过列
for (int i = 0; i < fec_index; ++i)
entry += entry_size_increment * (i + 1); // skip over the data.
// 确定目的掩码表的起始地址和size
size_t size = entry_size_increment * (fec_index + 1);
return {&entry[0], size};
}
2.2.4 webrtc中的Unequal Protection
非均等保护在webrtc中也实现了，但不是使用L1的方式，而采用了另一种方式，在指定的fec包数量下，分配更多的fec包给更重要的包，是通过掩码表的方式实现Unequal Protection的，这个东西实现在UnequalProtectionMask()
void UnequalProtectionMask(int num_media_packets,
int num_fec_packets,
int num_imp_packets,
int num_mask_bytes,
uint8_t* packet_mask,
PacketMaskTable* mask_table) {
// Set Protection type and allocation
// TODO(marpan): test/update for best mode and some combinations thereof.
ProtectionMode mode = kModeOverlap;
int num_fec_for_imp_packets = 0;
if (mode != kModeBiasFirstPacket) {
num_fec_for_imp_packets = SetProtectionAllocation(
num_media_packets, num_fec_packets, num_imp_packets);
}
int num_fec_remaining = num_fec_packets - num_fec_for_imp_packets;
// Done with setting protection type and allocation
//
// Generate sub_mask1
//
if (num_fec_for_imp_packets > 0) {
ImportantPacketProtection(num_fec_for_imp_packets, num_imp_packets,
num_mask_bytes, packet_mask, mask_table);
}
//
// Generate sub_mask2
//
if (num_fec_remaining > 0) {
RemainingPacketProtection(num_media_packets, num_fec_remaining,
num_fec_for_imp_packets, num_mask_bytes, mode,
packet_mask, mask_table);
}
}
这种unequal就主要体现在给重要保护的包分配多少fec包了，假设现在有m个原始包，n个fec包，m个原始包的前k个是需要重要保护的包，有三种分配方式:
kModeNoOverlap: 分配t个fec包给k个重要原始包做fec，剩下的fec包给剩下的普通原始包 (非重叠版)
kModeOverlap: 分配t个fec包给k个重要原始包做fec，剩下的fec包同时给普通版和重要包做fec (重叠版)
kModeBiasFirstPacket: 全部fec包都对第一个包做编码。
2.2.5 flexfec
ulpfec在某些丢包的情况下会导致无法恢复, 如下图所示，p2和p3如果在传输的过程中丢失了，就无法通过p1和f1去恢复了
此时如果再引入了列方向上的fec就能够比较好的解决这个问题,如下图，对列进行fec后，哪怕p2和p3同时丢失，
也能通过f5和f6对它进行复原, 这就是flexfec, 通过允许更直观灵活的组包fec应该是flexfec设计的目的，但丢包太多还是存在无法恢复的情况。
2.3 fec调整策略
fec的调整策略在细节上至今都没有看太明白，只能说一个大概；
fec_rate这个值为fec_packet_num / media_packet_num * 256，设定之后就能确定fec包的个数了，在2.2.3.2中也曾提到过。这个值是综合码率，丢包率得到的，具体的计算在VCMFecMethod::ProtectionFactor()中
bool VCMFecMethod::ProtectionFactor(const VCMProtectionParameters* parameters) {
// FEC PROTECTION SETTINGS: varies with packet loss and bitrate
// No protection if (filtered) packetLoss is 0
uint8_t packetLoss = rtc::saturated_cast<uint8_t>(255 * parameters->lossPr);
if (packetLoss == 0) {
_protectionFactorK = 0;
_protectionFactorD = 0;
return true;
}
// Parameters for FEC setting:
// first partition size, thresholds, table pars, spatial resoln fac.
// First partition protection: ~ 20%
uint8_t firstPartitionProt = rtc::saturated_cast<uint8_t>(255 * 0.20);
// Minimum protection level needed to generate one FEC packet for one
// source packet/frame (in RTP sender)
uint8_t minProtLevelFec = 85;
// Threshold on packetLoss and bitRrate/frameRate (=average #packets),
// above which we allocate protection to cover at least first partition.
uint8_t lossThr = 0;
uint8_t packetNumThr = 1;
// Parameters for range of rate index of table.
const uint8_t ratePar1 = 5;
const uint8_t ratePar2 = 49;
// webrtc提供了一个二维表kFecRateTable[rate][loss]，用于查询不同码率丢包下的
// fec_rate第一维是码率，第二维是丢包, 值是fec_rate
// Spatial resolution size, relative to a reference size.
// 计算当前分辨率和参考基分辨率 704 * 576的比
float spatialSizeToRef = rtc::saturated_cast<float>(parameters->codecWidth *
parameters->codecHeight) /
(rtc::saturated_cast<float>(704 * 576));
// resolnFac: This parameter will generally increase/decrease the FEC rate
// (for fixed bitRate and packetLoss) based on system size.
// Use a smaller exponent (< 1) to control/soften system size effect.
// 使用分辨率比得到一个变换系数，用于下文对实际分辨率的变换
// 使用这个变换系数是为了根据
const float resolnFac = 1.0 / powf(spatialSizeToRef, 0.3f);
// 通过parameter计算出每帧的码率
const int bitRatePerFrame = BitsPerFrame(parameters);
// Average number of packets per frame (source and fec):
const uint8_t avgTotPackets = rtc::saturated_cast<uint8_t>(
1.5f + rtc::saturated_cast<float>(bitRatePerFrame) * 1000.0f /
rtc::saturated_cast<float>(8.0 * _maxPayloadSize));
// FEC rate parameters: for P and I frame
uint8_t codeRateDelta = 0;
uint8_t codeRateKey = 0;
// Get index for table: the FEC protection depends on an effective rate.
// The range on the rate index corresponds to rates (bps)
// from ~200k to ~8000k, for 30fps
// 将码率进行变换，根据每一帧码率变换出表的行
const uint16_t effRateFecTable =
rtc::saturated_cast<uint16_t>(resolnFac * bitRatePerFrame);
uint8_t rateIndexTable = rtc::saturated_cast<uint8_t>(
VCM_MAX(VCM_MIN((effRateFecTable - ratePar1) / ratePar1, ratePar2), 0));
// Restrict packet loss range to 50:
// current tables defined only up to 50%
// 只支持50%的丢包
if (packetLoss >= kPacketLossMax) {
packetLoss = kPacketLossMax - 1;
}
// 计算查表得坐标
uint16_t indexTable = rateIndexTable * kPacketLossMax + packetLoss;
// Check on table index
RTC_DCHECK_LT(indexTable, kFecRateTableSize);
// Protection factor for P frame
// 从kFecRateTable[rateIndexTable][packetLoss] 查到对应的factor
codeRateDelta = kFecRateTable[indexTable];
if (packetLoss > lossThr && avgTotPackets > packetNumThr) {
// Set a minimum based on first partition size.
if (codeRateDelta < firstPartitionProt) {
codeRateDelta = firstPartitionProt;
}
}
// Check limit on amount of protection for P frame; 50% is max.
if (codeRateDelta >= kPacketLossMax) {
codeRateDelta = kPacketLossMax - 1;
}
// For Key frame:
// Effectively at a higher rate, so we scale/boost the rate
// The boost factor may depend on several factors: ratio of packet
// number of I to P frames, how much protection placed on P frames, etc.
// 实际码率上I帧大于P帧，所以会根据I、P帧中的packet数之比进行扩大
const uint8_t packetFrameDelta =
rtc::saturated_cast<uint8_t>(0.5 + parameters->packetsPerFrame);
const uint8_t packetFrameKey =
rtc::saturated_cast<uint8_t>(0.5 + parameters->packetsPerFrameKey);
const uint8_t boostKey = BoostCodeRateKey(packetFrameDelta, packetFrameKey);
rateIndexTable = rtc::saturated_cast<uint8_t>(VCM_MAX(
VCM_MIN(1 + (boostKey * effRateFecTable - ratePar1) / ratePar1, ratePar2),
0));
uint16_t indexTableKey = rateIndexTable * kPacketLossMax + packetLoss;
indexTableKey = VCM_MIN(indexTableKey, kFecRateTableSize);
// Check on table index
assert(indexTableKey < kFecRateTableSize);
// Protection factor for I frame
codeRateKey = kFecRateTable[indexTableKey];
// Boosting for Key frame.
int boostKeyProt = _scaleProtKey * codeRateDelta;
if (boostKeyProt >= kPacketLossMax) {
boostKeyProt = kPacketLossMax - 1;
}
// Make sure I frame protection is at least larger than P frame protection,
// and at least as high as filtered packet loss.
codeRateKey = rtc::saturated_cast<uint8_t>(
VCM_MAX(packetLoss, VCM_MAX(boostKeyProt, codeRateKey)));
// Check limit on amount of protection for I frame: 50% is max.
if (codeRateKey >= kPacketLossMax) {
codeRateKey = kPacketLossMax - 1;
}
// 设置I帧和P帧的fec rate
_protectionFactorK = codeRateKey;
_protectionFactorD = codeRateDelta;
// Generally there is a rate mis-match between the FEC cost estimated
// in mediaOpt and the actual FEC cost sent out in RTP module.
// This is more significant at low rates (small # of source packets), where
// the granularity of the FEC decreases. In this case, non-zero protection
// in mediaOpt may generate 0 FEC packets in RTP sender (since actual #FEC
// is based on rounding off protectionFactor on actual source packet number).
// The correction factor (_corrFecCost) attempts to corrects this, at least
// for cases of low rates (small #packets) and low protection levels.
// 由于fec rate的预估和实际的fec的使用存在出入，这一点在低码率的时候，fec rate
// 很小时影响很大，在这种情况下，预测一个非0的fec rate，可能导致rtp sender
// 产生-个fec包(因为fec包的实际计算会使用四舍五入)，所以引入了_corrFecCost 0.5
// 去做一个上修正
float numPacketsFl =
1.0f + (rtc::saturated_cast<float>(bitRatePerFrame) * 1000.0 /
rtc::saturated_cast<float>(8.0 * _maxPayloadSize) +
0.5);
const float estNumFecGen =
0.5f +
rtc::saturated_cast<float>(_protectionFactorD * numPacketsFl / 255.0f);
// We reduce cost factor (which will reduce overhead for FEC and
// hybrid method) and not the protectionFactor.
_corrFecCost = 1.0f;
// 对于单packet，预估系数小于产生一个fec包最小系数，预估产生的fec包小于[0.9, 1.1]，引入0.5上修正
if (estNumFecGen < 1.1f && _protectionFactorD < minProtLevelFec) {
_corrFecCost = 0.5f;
}
// 小于0.9 无需上修正
if (estNumFecGen < 0.9f && _protectionFactorD < minProtLevelFec) {
_corrFecCost = 0.0f;
}
// DONE WITH FEC PROTECTION SETTINGS
return true;
}
主要:
- webrtc提供了一个二维表kFecRateTable[rate][loss]，用于查询不同码率丢包下的fec_rate, 第一维是码率，第二维是丢包率, 值是fec_rate
- 计算P帧的码率，基于参考做一个变换(这个变换暂时理解不了)，根据这个变换后的码率作为第一码率，丢包率作为第二维查表得到fec_rate
- 鉴于I帧的码率比P帧大，使用I帧和P帧的packet数的倍数比，扩大P帧的第一维坐标作为I帧第一维坐标去查表，得到fec_rate
- 由于rtp module实际使用fec_rate对算出来的fec包数会做四舍五入(这点也很奇怪，应该是上取整)，在预估要产生一个fec包，但是预估的fec_rate无法无法达到单packet是最小能产生一个fec包的值，引入一个上修正系数_corrFecCost =0.5(但实际上是没用上的，因为rtp模块做的是上取整吧)
3.Ref
1.ULPFEC在WebRTC中的实现 -- weizhenwei
2.webrtc QOS方法二.2（ulpfec rfc5109简介）--CrystalShaw
5.RFC5109 RTP Payload Format for Generic Forward Error Correction
6. RTP Payload Format for Flexible Forward Error Correction (FlexFEC)