成功最有效的方法就是向有经验的人学习!

Kubernetes 部署 Ingress 控制器 Traefik v2.5

Traefik 简介

Traefik 是一款开源的边缘路由器,它可以让发布服务变得轻松有趣。它代表您的系统接收请求,并找出负责处理这些请求的组件。与众不同之处在于,除了它的许多特性之外,它还可以自动为您的服务发现正确的配置。当 Traefik 检查您的基础设施时,它会发现相关信息,并发现哪个服务为哪个请求提供服务。

Traefik 与每个主要的集群技术都是原生兼容的,比如 Kubernetes、Docker、Docker Swarm、AWS、Mesos、Marathon 等等;并且可以同时处理多个。(它甚至适用于运行在裸机上的遗留软件。)
使用 Traefik,不需要维护和同步单独的配置文件:所有事情都是实时自动发生的(没有重启,没有连接中断)。使用 Traefik,只需要花费时间开发和部署新功能到您的系统,而不是配置和维护其工作状态。

当部署完 Traefik 后还需要创建外部访问 Kubernetes 内部应用的路由规则,Traefik 目前支持三种方式创建路由规则方式:
一种是创建 Traefik 自定义 Kubernetes CRD 资源
另一种是创建 Kubernetes Ingress 资源
还有就是 v2.4 版本对 Kubernetes 扩展 API Kubernetes Gateway API 适配的一种方式,创建 GatewayClass、Gateway 与 HTTPRoute 资源。
我个人比较喜欢使用第一种,第三种方式目前还处理实验阶段

所以在我的环境中我只部署支持了CDR方式

创建 CRD 资源

# All resources definition must be declared

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: ingressroutes.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: IngressRoute
    listKind: IngressRouteList
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: IngressRoute is an Ingress CRD specification.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: IngressRouteSpec is a specification for a IngressRouteSpec
              resource.
            properties:
              entryPoints:
                items:
                  type: string
                type: array
              routes:
                items:
                  description: Route contains the set of routes.
                  properties:
                    kind:
                      enum:
                      - Rule
                      type: string
                    match:
                      type: string
                    middlewares:
                      items:
                        description: MiddlewareRef is a ref to the Middleware resources.
                        properties:
                          name:
                            type: string
                          namespace:
                            type: string
                        required:
                        - name
                        type: object
                      type: array
                    priority:
                      type: integer
                    services:
                      items:
                        description: Service defines an upstream to proxy traffic.
                        properties:
                          kind:
                            enum:
                            - Service
                            - TraefikService
                            type: string
                          name:
                            description: Name is a reference to a Kubernetes Service
                              object (for a load-balancer of servers), or to a TraefikService
                              object (service load-balancer, mirroring, etc). The
                              differentiation between the two is specified in the
                              Kind field.
                            type: string
                          namespace:
                            type: string
                          passHostHeader:
                            type: boolean
                          port:
                            anyOf:
                            - type: integer
                            - type: string
                            x-kubernetes-int-or-string: true
                          responseForwarding:
                            description: ResponseForwarding holds configuration for
                              the forward of the response.
                            properties:
                              flushInterval:
                                type: string
                            type: object
                          scheme:
                            type: string
                          serversTransport:
                            type: string
                          sticky:
                            description: Sticky holds the sticky configuration.
                            properties:
                              cookie:
                                description: Cookie holds the sticky configuration
                                  based on cookie.
                                properties:
                                  httpOnly:
                                    type: boolean
                                  name:
                                    type: string
                                  sameSite:
                                    type: string
                                  secure:
                                    type: boolean
                                type: object
                            type: object
                          strategy:
                            type: string
                          weight:
                            description: Weight should only be specified when Name
                              references a TraefikService object (and to be precise,
                              one that embeds a Weighted Round Robin).
                            type: integer
                        required:
                        - name
                        type: object
                      type: array
                  required:
                  - kind
                  - match
                  type: object
                type: array
              tls:
                description: "TLS contains the TLS certificates configuration of the
                  routes. To enable Let's Encrypt, use an empty TLS struct, e.g. in
                  YAML: \n \t tls: {} # inline format \n \t tls: \t   secretName:
                  # block format"
                properties:
                  certResolver:
                    type: string
                  domains:
                    items:
                      description: Domain holds a domain name with SANs.
                      properties:
                        main:
                          type: string
                        sans:
                          items:
                            type: string
                          type: array
                      type: object
                    type: array
                  options:
                    description: Options is a reference to a TLSOption, that specifies
                      the parameters of the TLS connection.
                    properties:
                      name:
                        type: string
                      namespace:
                        type: string
                    required:
                    - name
                    type: object
                  secretName:
                    description: SecretName is the name of the referenced Kubernetes
                      Secret to specify the certificate details.
                    type: string
                  store:
                    description: Store is a reference to a TLSStore, that specifies
                      the parameters of the TLS store.
                    properties:
                      name:
                        type: string
                      namespace:
                        type: string
                    required:
                    - name
                    type: object
                type: object
            required:
            - routes
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: ingressroutetcps.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: IngressRouteTCP
    listKind: IngressRouteTCPList
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: IngressRouteTCP is an Ingress CRD specification.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: IngressRouteTCPSpec is a specification for a IngressRouteTCPSpec
              resource.
            properties:
              entryPoints:
                items:
                  type: string
                type: array
              routes:
                items:
                  description: RouteTCP contains the set of routes.
                  properties:
                    match:
                      type: string
                    middlewares:
                      description: Middlewares contains references to MiddlewareTCP
                        resources.
                      items:
                        description: ObjectReference is a generic reference to a Traefik
                          resource.
                        properties:
                          name:
                            type: string
                          namespace:
                            type: string
                        required:
                        - name
                        type: object
                      type: array
                    services:
                      items:
                        description: ServiceTCP defines an upstream to proxy traffic.
                        properties:
                          name:
                            type: string
                          namespace:
                            type: string
                          port:
                            anyOf:
                            - type: integer
                            - type: string
                            x-kubernetes-int-or-string: true
                          proxyProtocol:
                            description: ProxyProtocol holds the ProxyProtocol configuration.
                            properties:
                              version:
                                type: integer
                            type: object
                          terminationDelay:
                            type: integer
                          weight:
                            type: integer
                        required:
                        - name
                        - port
                        type: object
                      type: array
                  required:
                  - match
                  type: object
                type: array
              tls:
                description: "TLSTCP contains the TLS certificates configuration of
                  the routes. To enable Let's Encrypt, use an empty TLS struct, e.g.
                  in YAML: \n \t tls: {} # inline format \n \t tls: \t   secretName:
                  # block format"
                properties:
                  certResolver:
                    type: string
                  domains:
                    items:
                      description: Domain holds a domain name with SANs.
                      properties:
                        main:
                          type: string
                        sans:
                          items:
                            type: string
                          type: array
                      type: object
                    type: array
                  options:
                    description: Options is a reference to a TLSOption, that specifies
                      the parameters of the TLS connection.
                    properties:
                      name:
                        type: string
                      namespace:
                        type: string
                    required:
                    - name
                    type: object
                  passthrough:
                    type: boolean
                  secretName:
                    description: SecretName is the name of the referenced Kubernetes
                      Secret to specify the certificate details.
                    type: string
                  store:
                    description: Store is a reference to a TLSStore, that specifies
                      the parameters of the TLS store.
                    properties:
                      name:
                        type: string
                      namespace:
                        type: string
                    required:
                    - name
                    type: object
                type: object
            required:
            - routes
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: ingressrouteudps.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: IngressRouteUDP
    listKind: IngressRouteUDPList
    plural: ingressrouteudps
    singular: ingressrouteudp
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: IngressRouteUDP is an Ingress CRD specification.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: IngressRouteUDPSpec is a specification for a IngressRouteUDPSpec
              resource.
            properties:
              entryPoints:
                items:
                  type: string
                type: array
              routes:
                items:
                  description: RouteUDP contains the set of routes.
                  properties:
                    services:
                      items:
                        description: ServiceUDP defines an upstream to proxy traffic.
                        properties:
                          name:
                            type: string
                          namespace:
                            type: string
                          port:
                            anyOf:
                            - type: integer
                            - type: string
                            x-kubernetes-int-or-string: true
                          weight:
                            type: integer
                        required:
                        - name
                        - port
                        type: object
                      type: array
                  type: object
                type: array
            required:
            - routes
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: middlewares.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: Middleware
    listKind: MiddlewareList
    plural: middlewares
    singular: middleware
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: Middleware is a specification for a Middleware resource.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: MiddlewareSpec holds the Middleware configuration.
            properties:
              addPrefix:
                description: AddPrefix holds the AddPrefix configuration.
                properties:
                  prefix:
                    type: string
                type: object
              basicAuth:
                description: BasicAuth holds the HTTP basic authentication configuration.
                properties:
                  headerField:
                    type: string
                  realm:
                    type: string
                  removeHeader:
                    type: boolean
                  secret:
                    type: string
                type: object
              buffering:
                description: Buffering holds the request/response buffering configuration.
                properties:
                  maxRequestBodyBytes:
                    format: int64
                    type: integer
                  maxResponseBodyBytes:
                    format: int64
                    type: integer
                  memRequestBodyBytes:
                    format: int64
                    type: integer
                  memResponseBodyBytes:
                    format: int64
                    type: integer
                  retryExpression:
                    type: string
                type: object
              chain:
                description: Chain holds a chain of middlewares.
                properties:
                  middlewares:
                    items:
                      description: MiddlewareRef is a ref to the Middleware resources.
                      properties:
                        name:
                          type: string
                        namespace:
                          type: string
                      required:
                      - name
                      type: object
                    type: array
                type: object
              circuitBreaker:
                description: CircuitBreaker holds the circuit breaker configuration.
                properties:
                  expression:
                    type: string
                type: object
              compress:
                description: Compress holds the compress configuration.
                properties:
                  excludedContentTypes:
                    items:
                      type: string
                    type: array
                type: object
              contentType:
                description: ContentType middleware - or rather its unique `autoDetect`
                  option - specifies whether to let the `Content-Type` header, if
                  it has not been set by the backend, be automatically set to a value
                  derived from the contents of the response. As a proxy, the default
                  behavior should be to leave the header alone, regardless of what
                  the backend did with it. However, the historic default was to always
                  auto-detect and set the header if it was nil, and it is going to
                  be kept that way in order to support users currently relying on
                  it. This middleware exists to enable the correct behavior until
                  at least the default one can be changed in a future version.
                properties:
                  autoDetect:
                    type: boolean
                type: object
              digestAuth:
                description: DigestAuth holds the Digest HTTP authentication configuration.
                properties:
                  headerField:
                    type: string
                  realm:
                    type: string
                  removeHeader:
                    type: boolean
                  secret:
                    type: string
                type: object
              errors:
                description: ErrorPage holds the custom error page configuration.
                properties:
                  query:
                    type: string
                  service:
                    description: Service defines an upstream to proxy traffic.
                    properties:
                      kind:
                        enum:
                        - Service
                        - TraefikService
                        type: string
                      name:
                        description: Name is a reference to a Kubernetes Service object
                          (for a load-balancer of servers), or to a TraefikService
                          object (service load-balancer, mirroring, etc). The differentiation
                          between the two is specified in the Kind field.
                        type: string
                      namespace:
                        type: string
                      passHostHeader:
                        type: boolean
                      port:
                        anyOf:
                        - type: integer
                        - type: string
                        x-kubernetes-int-or-string: true
                      responseForwarding:
                        description: ResponseForwarding holds configuration for the
                          forward of the response.
                        properties:
                          flushInterval:
                            type: string
                        type: object
                      scheme:
                        type: string
                      serversTransport:
                        type: string
                      sticky:
                        description: Sticky holds the sticky configuration.
                        properties:
                          cookie:
                            description: Cookie holds the sticky configuration based
                              on cookie.
                            properties:
                              httpOnly:
                                type: boolean
                              name:
                                type: string
                              sameSite:
                                type: string
                              secure:
                                type: boolean
                            type: object
                        type: object
                      strategy:
                        type: string
                      weight:
                        description: Weight should only be specified when Name references
                          a TraefikService object (and to be precise, one that embeds
                          a Weighted Round Robin).
                        type: integer
                    required:
                    - name
                    type: object
                  status:
                    items:
                      type: string
                    type: array
                type: object
              forwardAuth:
                description: ForwardAuth holds the http forward authentication configuration.
                properties:
                  address:
                    type: string
                  authRequestHeaders:
                    items:
                      type: string
                    type: array
                  authResponseHeaders:
                    items:
                      type: string
                    type: array
                  authResponseHeadersRegex:
                    type: string
                  tls:
                    description: ClientTLS holds TLS specific configurations as client.
                    properties:
                      caOptional:
                        type: boolean
                      caSecret:
                        type: string
                      certSecret:
                        type: string
                      insecureSkipVerify:
                        type: boolean
                    type: object
                  trustForwardHeader:
                    type: boolean
                type: object
              headers:
                description: Headers holds the custom header configuration.
                properties:
                  accessControlAllowCredentials:
                    description: AccessControlAllowCredentials is only valid if true.
                      false is ignored.
                    type: boolean
                  accessControlAllowHeaders:
                    description: AccessControlAllowHeaders must be used in response
                      to a preflight request with Access-Control-Request-Headers set.
                    items:
                      type: string
                    type: array
                  accessControlAllowMethods:
                    description: AccessControlAllowMethods must be used in response
                      to a preflight request with Access-Control-Request-Method set.
                    items:
                      type: string
                    type: array
                  accessControlAllowOriginList:
                    description: AccessControlAllowOriginList is a list of allowable
                      origins. Can also be a wildcard origin "*".
                    items:
                      type: string
                    type: array
                  accessControlAllowOriginListRegex:
                    description: AccessControlAllowOriginListRegex is a list of allowable
                      origins written following the Regular Expression syntax (https://golang.org/pkg/regexp/).
                    items:
                      type: string
                    type: array
                  accessControlExposeHeaders:
                    description: AccessControlExposeHeaders sets valid headers for
                      the response.
                    items:
                      type: string
                    type: array
                  accessControlMaxAge:
                    description: AccessControlMaxAge sets the time that a preflight
                      request may be cached.
                    format: int64
                    type: integer
                  addVaryHeader:
                    description: AddVaryHeader controls if the Vary header is automatically
                      added/updated when the AccessControlAllowOriginList is set.
                    type: boolean
                  allowedHosts:
                    items:
                      type: string
                    type: array
                  browserXssFilter:
                    type: boolean
                  contentSecurityPolicy:
                    type: string
                  contentTypeNosniff:
                    type: boolean
                  customBrowserXSSValue:
                    type: string
                  customFrameOptionsValue:
                    type: string
                  customRequestHeaders:
                    additionalProperties:
                      type: string
                    type: object
                  customResponseHeaders:
                    additionalProperties:
                      type: string
                    type: object
                  featurePolicy:
                    description: 'Deprecated: use PermissionsPolicy instead.'
                    type: string
                  forceSTSHeader:
                    type: boolean
                  frameDeny:
                    type: boolean
                  hostsProxyHeaders:
                    items:
                      type: string
                    type: array
                  isDevelopment:
                    type: boolean
                  permissionsPolicy:
                    type: string
                  publicKey:
                    type: string
                  referrerPolicy:
                    type: string
                  sslForceHost:
                    description: 'Deprecated: use RedirectRegex instead.'
                    type: boolean
                  sslHost:
                    description: 'Deprecated: use RedirectRegex instead.'
                    type: string
                  sslProxyHeaders:
                    additionalProperties:
                      type: string
                    type: object
                  sslRedirect:
                    description: 'Deprecated: use EntryPoint redirection or RedirectScheme
                      instead.'
                    type: boolean
                  sslTemporaryRedirect:
                    description: 'Deprecated: use EntryPoint redirection or RedirectScheme
                      instead.'
                    type: boolean
                  stsIncludeSubdomains:
                    type: boolean
                  stsPreload:
                    type: boolean
                  stsSeconds:
                    format: int64
                    type: integer
                type: object
              inFlightReq:
                description: InFlightReq limits the number of requests being processed
                  and served concurrently.
                properties:
                  amount:
                    format: int64
                    type: integer
                  sourceCriterion:
                    description: SourceCriterion defines what criterion is used to
                      group requests as originating from a common source. If none
                      are set, the default is to use the request's remote address
                      field. All fields are mutually exclusive.
                    properties:
                      ipStrategy:
                        description: IPStrategy holds the ip strategy configuration.
                        properties:
                          depth:
                            type: integer
                          excludedIPs:
                            items:
                              type: string
                            type: array
                        type: object
                      requestHeaderName:
                        type: string
                      requestHost:
                        type: boolean
                    type: object
                type: object
              ipWhiteList:
                description: IPWhiteList holds the ip white list configuration.
                properties:
                  ipStrategy:
                    description: IPStrategy holds the ip strategy configuration.
                    properties:
                      depth:
                        type: integer
                      excludedIPs:
                        items:
                          type: string
                        type: array
                    type: object
                  sourceRange:
                    items:
                      type: string
                    type: array
                type: object
              passTLSClientCert:
                description: PassTLSClientCert holds the TLS client cert headers configuration.
                properties:
                  info:
                    description: TLSClientCertificateInfo holds the client TLS certificate
                      info configuration.
                    properties:
                      issuer:
                        description: TLSClientCertificateDNInfo holds the client TLS
                          certificate distinguished name info configuration. cf https://tools.ietf.org/html/rfc3739
                        properties:
                          commonName:
                            type: boolean
                          country:
                            type: boolean
                          domainComponent:
                            type: boolean
                          locality:
                            type: boolean
                          organization:
                            type: boolean
                          province:
                            type: boolean
                          serialNumber:
                            type: boolean
                        type: object
                      notAfter:
                        type: boolean
                      notBefore:
                        type: boolean
                      sans:
                        type: boolean
                      serialNumber:
                        type: boolean
                      subject:
                        description: TLSClientCertificateDNInfo holds the client TLS
                          certificate distinguished name info configuration. cf https://tools.ietf.org/html/rfc3739
                        properties:
                          commonName:
                            type: boolean
                          country:
                            type: boolean
                          domainComponent:
                            type: boolean
                          locality:
                            type: boolean
                          organization:
                            type: boolean
                          province:
                            type: boolean
                          serialNumber:
                            type: boolean
                        type: object
                    type: object
                  pem:
                    type: boolean
                type: object
              plugin:
                additionalProperties:
                  x-kubernetes-preserve-unknown-fields: true
                type: object
              rateLimit:
                description: RateLimit holds the rate limiting configuration for a
                  given router.
                properties:
                  average:
                    format: int64
                    type: integer
                  burst:
                    format: int64
                    type: integer
                  period:
                    anyOf:
                    - type: integer
                    - type: string
                    x-kubernetes-int-or-string: true
                  sourceCriterion:
                    description: SourceCriterion defines what criterion is used to
                      group requests as originating from a common source. If none
                      are set, the default is to use the request's remote address
                      field. All fields are mutually exclusive.
                    properties:
                      ipStrategy:
                        description: IPStrategy holds the ip strategy configuration.
                        properties:
                          depth:
                            type: integer
                          excludedIPs:
                            items:
                              type: string
                            type: array
                        type: object
                      requestHeaderName:
                        type: string
                      requestHost:
                        type: boolean
                    type: object
                type: object
              redirectRegex:
                description: RedirectRegex holds the redirection configuration.
                properties:
                  permanent:
                    type: boolean
                  regex:
                    type: string
                  replacement:
                    type: string
                type: object
              redirectScheme:
                description: RedirectScheme holds the scheme redirection configuration.
                properties:
                  permanent:
                    type: boolean
                  port:
                    type: string
                  scheme:
                    type: string
                type: object
              replacePath:
                description: ReplacePath holds the ReplacePath configuration.
                properties:
                  path:
                    type: string
                type: object
              replacePathRegex:
                description: ReplacePathRegex holds the ReplacePathRegex configuration.
                properties:
                  regex:
                    type: string
                  replacement:
                    type: string
                type: object
              retry:
                description: Retry holds the retry configuration.
                properties:
                  attempts:
                    type: integer
                  initialInterval:
                    anyOf:
                    - type: integer
                    - type: string
                    x-kubernetes-int-or-string: true
                type: object
              stripPrefix:
                description: StripPrefix holds the StripPrefix configuration.
                properties:
                  forceSlash:
                    type: boolean
                  prefixes:
                    items:
                      type: string
                    type: array
                type: object
              stripPrefixRegex:
                description: StripPrefixRegex holds the StripPrefixRegex configuration.
                properties:
                  regex:
                    items:
                      type: string
                    type: array
                type: object
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: middlewaretcps.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: MiddlewareTCP
    listKind: MiddlewareTCPList
    plural: middlewaretcps
    singular: middlewaretcp
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: MiddlewareTCP is a specification for a MiddlewareTCP resource.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: MiddlewareTCPSpec holds the MiddlewareTCP configuration.
            properties:
              ipWhiteList:
                description: TCPIPWhiteList holds the TCP ip white list configuration.
                properties:
                  sourceRange:
                    items:
                      type: string
                    type: array
                type: object
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: serverstransports.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: ServersTransport
    listKind: ServersTransportList
    plural: serverstransports
    singular: serverstransport
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: ServersTransport is a specification for a ServersTransport resource.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: ServersTransportSpec options to configure communication between
              Traefik and the servers.
            properties:
              certificatesSecrets:
                description: Certificates for mTLS.
                items:
                  type: string
                type: array
              disableHTTP2:
                description: Disable HTTP/2 for connections with backend servers.
                type: boolean
              forwardingTimeouts:
                description: Timeouts for requests forwarded to the backend servers.
                properties:
                  dialTimeout:
                    anyOf:
                    - type: integer
                    - type: string
                    description: The amount of time to wait until a connection to
                      a backend server can be established. If zero, no timeout exists.
                    x-kubernetes-int-or-string: true
                  idleConnTimeout:
                    anyOf:
                    - type: integer
                    - type: string
                    description: The maximum period for which an idle HTTP keep-alive
                      connection will remain open before closing itself.
                    x-kubernetes-int-or-string: true
                  responseHeaderTimeout:
                    anyOf:
                    - type: integer
                    - type: string
                    description: The amount of time to wait for a server's response
                      headers after fully writing the request (including its body,
                      if any). If zero, no timeout exists.
                    x-kubernetes-int-or-string: true
                type: object
              insecureSkipVerify:
                description: Disable SSL certificate verification.
                type: boolean
              maxIdleConnsPerHost:
                description: If non-zero, controls the maximum idle (keep-alive) to
                  keep per-host. If zero, DefaultMaxIdleConnsPerHost is used.
                type: integer
              peerCertURI:
                description: URI used to match against SAN URI during the peer certificate
                  verification.
                type: string
              rootCAsSecrets:
                description: Add cert file for self-signed certificate.
                items:
                  type: string
                type: array
              serverName:
                description: ServerName used to contact the server.
                type: string
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: tlsoptions.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: TLSOption
    listKind: TLSOptionList
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: TLSOption is a specification for a TLSOption resource.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TLSOptionSpec configures TLS for an entry point.
            properties:
              alpnProtocols:
                items:
                  type: string
                type: array
              cipherSuites:
                items:
                  type: string
                type: array
              clientAuth:
                description: ClientAuth defines the parameters of the client authentication
                  part of the TLS connection, if any.
                properties:
                  clientAuthType:
                    description: ClientAuthType defines the client authentication
                      type to apply.
                    enum:
                    - NoClientCert
                    - RequestClientCert
                    - RequireAnyClientCert
                    - VerifyClientCertIfGiven
                    - RequireAndVerifyClientCert
                    type: string
                  secretNames:
                    description: SecretName is the name of the referenced Kubernetes
                      Secret to specify the certificate details.
                    items:
                      type: string
                    type: array
                type: object
              curvePreferences:
                items:
                  type: string
                type: array
              maxVersion:
                type: string
              minVersion:
                type: string
              preferServerCipherSuites:
                type: boolean
              sniStrict:
                type: boolean
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: tlsstores.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: TLSStore
    listKind: TLSStoreList
    plural: tlsstores
    singular: tlsstore
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: TLSStore is a specification for a TLSStore resource.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TLSStoreSpec configures a TLSStore resource.
            properties:
              defaultCertificate:
                description: DefaultCertificate holds a secret name for the TLSOption
                  resource.
                properties:
                  secretName:
                    description: SecretName is the name of the referenced Kubernetes
                      Secret to specify the certificate details.
                    type: string
                required:
                - secretName
                type: object
            required:
            - defaultCertificate
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: traefikservices.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: TraefikService
    listKind: TraefikServiceList
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: TraefikService is the specification for a service (that an IngressRoute
          refers to) that is usually not a terminal service (i.e. not a pod of servers),
          as opposed to a Kubernetes Service. That is to say, it usually refers to
          other (children) services, which themselves can be TraefikServices or Services.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: ServiceSpec defines whether a TraefikService is a load-balancer
              of services or a mirroring service.
            properties:
              mirroring:
                description: Mirroring defines a mirroring service, which is composed
                  of a main load-balancer, and a list of mirrors.
                properties:
                  kind:
                    enum:
                    - Service
                    - TraefikService
                    type: string
                  maxBodySize:
                    format: int64
                    type: integer
                  mirrors:
                    items:
                      description: MirrorService defines one of the mirrors of a Mirroring
                        service.
                      properties:
                        kind:
                          enum:
                          - Service
                          - TraefikService
                          type: string
                        name:
                          description: Name is a reference to a Kubernetes Service
                            object (for a load-balancer of servers), or to a TraefikService
                            object (service load-balancer, mirroring, etc). The differentiation
                            between the two is specified in the Kind field.
                          type: string
                        namespace:
                          type: string
                        passHostHeader:
                          type: boolean
                        percent:
                          type: integer
                        port:
                          anyOf:
                          - type: integer
                          - type: string
                          x-kubernetes-int-or-string: true
                        responseForwarding:
                          description: ResponseForwarding holds configuration for
                            the forward of the response.
                          properties:
                            flushInterval:
                              type: string
                          type: object
                        scheme:
                          type: string
                        serversTransport:
                          type: string
                        sticky:
                          description: Sticky holds the sticky configuration.
                          properties:
                            cookie:
                              description: Cookie holds the sticky configuration based
                                on cookie.
                              properties:
                                httpOnly:
                                  type: boolean
                                name:
                                  type: string
                                sameSite:
                                  type: string
                                secure:
                                  type: boolean
                              type: object
                          type: object
                        strategy:
                          type: string
                        weight:
                          description: Weight should only be specified when Name references
                            a TraefikService object (and to be precise, one that embeds
                            a Weighted Round Robin).
                          type: integer
                      required:
                      - name
                      type: object
                    type: array
                  name:
                    description: Name is a reference to a Kubernetes Service object
                      (for a load-balancer of servers), or to a TraefikService object
                      (service load-balancer, mirroring, etc). The differentiation
                      between the two is specified in the Kind field.
                    type: string
                  namespace:
                    type: string
                  passHostHeader:
                    type: boolean
                  port:
                    anyOf:
                    - type: integer
                    - type: string
                    x-kubernetes-int-or-string: true
                  responseForwarding:
                    description: ResponseForwarding holds configuration for the forward
                      of the response.
                    properties:
                      flushInterval:
                        type: string
                    type: object
                  scheme:
                    type: string
                  serversTransport:
                    type: string
                  sticky:
                    description: Sticky holds the sticky configuration.
                    properties:
                      cookie:
                        description: Cookie holds the sticky configuration based on
                          cookie.
                        properties:
                          httpOnly:
                            type: boolean
                          name:
                            type: string
                          sameSite:
                            type: string
                          secure:
                            type: boolean
                        type: object
                    type: object
                  strategy:
                    type: string
                  weight:
                    description: Weight should only be specified when Name references
                      a TraefikService object (and to be precise, one that embeds
                      a Weighted Round Robin).
                    type: integer
                required:
                - name
                type: object
              weighted:
                description: WeightedRoundRobin defines a load-balancer of services.
                properties:
                  services:
                    items:
                      description: Service defines an upstream to proxy traffic.
                      properties:
                        kind:
                          enum:
                          - Service
                          - TraefikService
                          type: string
                        name:
                          description: Name is a reference to a Kubernetes Service
                            object (for a load-balancer of servers), or to a TraefikService
                            object (service load-balancer, mirroring, etc). The differentiation
                            between the two is specified in the Kind field.
                          type: string
                        namespace:
                          type: string
                        passHostHeader:
                          type: boolean
                        port:
                          anyOf:
                          - type: integer
                          - type: string
                          x-kubernetes-int-or-string: true
                        responseForwarding:
                          description: ResponseForwarding holds configuration for
                            the forward of the response.
                          properties:
                            flushInterval:
                              type: string
                          type: object
                        scheme:
                          type: string
                        serversTransport:
                          type: string
                        sticky:
                          description: Sticky holds the sticky configuration.
                          properties:
                            cookie:
                              description: Cookie holds the sticky configuration based
                                on cookie.
                              properties:
                                httpOnly:
                                  type: boolean
                                name:
                                  type: string
                                sameSite:
                                  type: string
                                secure:
                                  type: boolean
                              type: object
                          type: object
                        strategy:
                          type: string
                        weight:
                          description: Weight should only be specified when Name references
                            a TraefikService object (and to be precise, one that embeds
                            a Weighted Round Robin).
                          type: integer
                      required:
                      - name
                      type: object
                    type: array
                  sticky:
                    description: Sticky holds the sticky configuration.
                    properties:
                      cookie:
                        description: Cookie holds the sticky configuration based on
                          cookie.
                        properties:
                          httpOnly:
                            type: boolean
                          name:
                            type: string
                          sameSite:
                            type: string
                          secure:
                            type: boolean
                        type: object
                    type: object
                type: object
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

创建 RBAC 权限

Kubernetes 在 1.6 版本中引入了基于角色的访问控制(RBAC)策略,方便对 Kubernetes 资源和 API 进行细粒度控制。Traefik 需要一定的权限,所以,这里提前创建好 Traefik ServiceAccount 并分配一定的权限。

kind: ServiceAccount
apiVersion: v1
metadata:
  name: traefik-ingress-controller
  namespace: ingress-traefik
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
      - middlewaretcps
      - ingressroutes
      - traefikservices
      - ingressroutetcps
      - ingressrouteudps
      - tlsoptions
      - tlsstores
      - serverstransports
    verbs:
      - get
      - list
      - watch

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: traefik-ingress-controller

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: ingress-traefik

创建 Traefik 配置文件

由于 Traefik 配置很多,通过 CLI 定义不是很方便,一般时候都会通过配置文件配置 Traefik 参数,然后存入 ConfigMap,将其挂入 Traefik 中。

创建 traefik-config.yaml 文件

kind: ConfigMap
apiVersion: v1
metadata:
  name: traefik-config
  namespace: ingress-traefik
data:
  traefik.yaml: |-
    ping: ""          
    serversTransport:
      insecureSkipVerify: true  
    api:
      insecure: true            
      dashboard: true  
      debug: false             
    metrics:
      prometheus:
        entrypoint: "metrics"      
        addEntryPointsLabels: true
        addRoutersLabels: true
        addServicesLabels: true
    entryPoints:
      web:
        address: ":80"  
      websecure:
        address: ":443"      
      metrics:
        address: ":9100"
      traefik:
        address: ":9000"
    providers:
      kubernetesCRD:
        allowCrossNamespace: true
        allowExternalNameServices: true        
      kubernetesIngress: ""
      #kubernetesGateway: ""
    experimental:
      kubernetesGateway: true
    log:
      filePath: ""            
      level: error       
      format: json        
    accessLog:
      filePath: ""       
      format: json             
      bufferingSize: 0       
      filters:
        #statusCodes: ["200"]   
        retryAttempts: true     
        minDuration: 20        
      fields:               
        defaultMode: keep 
        names:                  
          ClientUsername: drop  
        headers:               
          defaultMode: keep     
          names:                
            User-Agent: redact
            Authorization: drop
            Content-Type: keep
    #tracing:             
    #  serviceName:               
    #  zipkin:             
    #    sameSpan: true        
    #    id128Bit: true     
    #    sampleRate: 0.1       
    #    httpEndpoint: http://localhost:9411/api/v2/spans

下面配置中可以通过配置 kubernetesCRD 与 kubernetesIngress 和 kubernetesGateway 三项参数,让 Traefik 支持 CRD、Ingress 与 kubernetesGateway 三种路由配置方式。

节点设置 Label 标签

由于是 Kubernetes DeamonSet 这种方式部署 Traefik,所以需要提前给节点设置 Label,这样当程序部署时会自动调度到设置 Label 的节点上。

格式:kubectl label nodes [节点名] [key=value]

kubectl label nodes k8s-node2 IngressProxy=true

如果想删除标签,可以使用 "kubectl label nodes k8s-node2 IngressProxy-" 命令

安装 Kubernetes Gateway CRD 资源

本例略
参照:https://doc.traefik.io/traefik/providers/kubernetes-gateway/

Kubernetes 部署 Traefik

下面将用 DaemonSet 方式部署 Traefik,便于在多服务器间扩展,用 hostport 方式绑定服务器 80、443 端口,方便流量通过物理机进入 Kubernetes 内部。

---
# Source: traefik/templates/dashboard-hook-ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: ingress-traefik
  labels:
    app.kubernetes.io/name: traefik
    app.kubernetes.io/instance: traefik
spec:
  entryPoints:
    - traefik
  routes:
  - match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
    kind: Rule
    services:
    - name: api@internal
      kind: TraefikService
MANIFEST:
---
您暂时无权查看此隐藏内容!

file
至此部署完成

简单使用示例

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: kubernetes-dashboard-route
  namespace: kubernetes-dashboard
spec:
  entryPoints:
  - websecure
  tls:
    secretName: uzgood-tls
  routes:
  - match: Host(`k8s.uzigood.tech`) 
    kind: Rule
    services:
      - name: kubernetes-dashboard
        port: 443
        namespace: kubernetes-dashboard
    middlewares:
    - name: flask-k8s-traffic
    - name: flask-k8s-1
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard-route1
  namespace: kubernetes-dashboard
spec:
  entryPoints:
  - web
  routes:
  - match: Host(`k8s.uzigood.tech`)
    kind: Rule
    middlewares:
    - name: test-redirectscheme
    services:
      - name: kubernetes-dashboard
        port: 443
        namespace: kubernetes-dashboard

关于证书管理,注意文件名称

kubectl create secret generic uzgood-tls --from-file=tls.crt --from-file=tls.key -n ingress-traefik

关于限流熔断跳转和中间件
Traefik Middlewares 是一个处于路由和后端服务之前的中间件,在外部流量进入 Traefik,且路由规则匹配成功后,将流量发送到对应的后端服务前,先将其发给中间件进行一些列处理(类似于过滤器链 Filter,进行一系列处理),例如,添加 Header 头信息、鉴权、流量转发、处理访问路径前缀、IP 白名单等等,经过一个或者多个中间件处理完成后,再发送给后端服务,这个就是中间件的作用。

# 断路器
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
您暂时无权查看此隐藏内容!

circuitBreaker(熔断)三个触发器:

  • The network error ratio (NetworkErrorRatio)
  • The status code ratio (ResponseCodeRatio)
  • The latency at quantile, in milliseconds (LatencyAtQuantileMS

NetworkErrorRatio
如果您希望断路器以 30% 的网络错误比率触发,则表示方式将是NetworkErrorRatio() > 0.30

ResponseCodeRatio
您可以根据给定状态代码范围的比例触发断路器。

接受四个参数。 from to dividedByFrom dividedByTo

将要计算的操作是 sum( to-> from) / sum ( dividedByFrom-> dividedByTo)

如果 sum ( dividedByFrom-> dividedByTo) 等于 0,则ResponseCodeRatio返回 0

from是包容的,to是排他的。

例如,ResponseCodeRatio(500, 600, 0, 600) > 0.25 如果 25% 的请求返回 5XX 状态(在返回状态代码从 0 到 5XX 的请求中),则表达式将触发断路器。

LatencyAtQuantileMS
当给定比例的请求变得太慢时,您可以触发断路器。

例如,当中位延迟(量级 50)达到 100MS 时,该表示将触发断路器。LatencyAtQuantileMS(50.0) > 100

您必须提供一个浮动号(尾随 .0)的量化值

使用多个指标
您可以使用操作员在表达中组合多个指标。

支持的操作员包括:

和 (&&)
或 (||)
例如,当 30% 的请求返回 5XX 状态代码时,或者当网络错误比率达到 10% 时触发断路器。ResponseCodeRatio(500, 600, 0, 600) > 0.30 || NetworkErrorRatio() > 0.10

VIP学员请参考内部文库
https://docs.ct99.cn/web/#/166?page_id=1398

内容查看本文隐藏内容查看需要消耗20土豆币,请先
土豆币按需购买,不退换,请考虑清楚后购买。
赞(4) 打赏
未经允许不得转载:陈桂林博客 » Kubernetes 部署 Ingress 控制器 Traefik v2.5

大佬们的评论 5

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
  1. #5

    google搜索资料到,对比哈。

    snake4个月前 (03-07)回复
  2. #4

    正在学习中 多谢分享

    1234个月前 (02-28)回复
  3. #3

    正在学习中 多谢分享

    Hitao5个月前 (01-15)回复
  4. #2

    正在学习中 多谢分享

    cnak476个月前 (01-09)回复
  5. #1

    非常感谢分享

    tzjs6个月前 (12-28)回复

全新“一站式”建站,高质量、高售后的一条龙服务

橙子建站.极速智能建站8折购买虚拟主机

觉得文章有用就打赏一下文章作者

非常感谢你的打赏,我们将继续给力更多优质内容,让我们一起创建更加美好的网络世界!

支付宝扫一扫打赏

微信扫一扫打赏